Opened 4 years ago

Last modified 2 hours ago

#26511 new enhancement

Meta-ticket: Use Python optimization interfaces: CVXPY, or-tools, PuLP, Pyomo, cylp...

Reported by: mkoeppe Owned by:
Priority: major Milestone: sage-9.8
Component: numerical Keywords:
Cc: a.mason@…, gh-jiawei-wang-ucd, yzh, dimpase, dcoudert, tmonteil, fbissey Merged in:
Authors: Reviewers:
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by mkoeppe)

The purpose of this ticket is to

  • connect SageMath to interfaces to optimization solvers that are maintained outside of the Sage project,
  • integrate the related developer and user communities,
  • provide an entry point for GSoC 2022 projects

SageMath status quo: Front end

  • Frontend class MixedIntegerLinearProgram
    • mutable (can call add_constraint, set_integer, new_variable etc. and then re-solve)
    • solver-independent and solver-specific parameters (solver_parameter)
    • widely used in Sage library code for sage.graphs, sage.coding, sage.combinat, ...
    • MIPVariable - indexed by arbitrary objects
    • some connections to Polyhedron class and to InteractiveLPProblem (didactical code)

SageMath status quo: Back ends

SageMath tickets and tasks

  • #28175 Move sage optimization backends to separate Cython packages to remove OptionalExtension problems
  • Template for minimal Python-based backends (interactivelp_backend.pyx is implemented in Cython just because a Cython template was available)
  • Replace basic Cython-based COIN backend by full-featured PuLP backend (gives access to many solvers)
  • #19219 MILP: Add CyLP backend -- Replace basic Cython-based COIN backend by full-featured cylp backend (gives detailed access to CLP, including tableau data)
  • PuLP backend implementation using Sage backends... to make exact backends such as InteractiveLP, PPL backend available to PuLP frontend users
    • can PuLP handle non-floating point data (for example for exact computation over rationals or number fields if the backend allows it?
    • this could be added to PuLP, rather than to Sage.
  • Pyomo backend implementation using Sage backends... to make exact backends available to Pyomo frontend users
    • see above
  • Check feasibility of replacing front end MixedIntegerLinearProgram by PuLP frontend or Pyomo frontend
    • can PuLP handle Sage's number types such as RDF and ZZ?
    • can Pyomo handle Sage's number types such as RDF and ZZ?

Related:

  • #20302 Meta-ticket: Improvements to MixedIntegerLinearProgram, its backends, and InteractiveLinearProgram

Key Python software (solver-independent)

Selection criteria for solver-independent interfaces:

  • Does it use a compatible, permissive, open source license?
  • Does it provide detailed access to the solver that allows for updating problems and warmstarting?
  • Is it under active development?

cvxpy - Meta-ticket #33920

or-tools - #33493

  • Upcoming ("soon" as of 2022-08) new MathOpt DSL (Python)

conv_opt - https://github.com/KarrLab/conv_opt

  • MIT license
  • Cbc, CVXOPT, FICO XPRESS, GLPK, Gurobi, IBM CPLEX, MINOS, Mosek, quadprog, SciPy, and SoPlex.

PuLP - https://github.com/coin-or/pulp

  • permissive open source license (BSD?)
  • Installation with sage -pip install pulp works

Pyomo

  • permissive open source license: BSD
  • http://www.pyomo.org/
  • installation with sage -pip install pyomo works
  • some file-based, some API-based interfaces (direct/persistent) - see solvers directory
  • big system, under active development

YAPOSIB - Python interface to COIN OSI, using Boost::Python

  • https://github.com/coin-or/yaposib
  • Installation with sage -i boost && sage -pip install yaposib works
  • incompatible open source license - EPL 1.0
  • therefore we cannot use it as the basis for our solver-independent code
  • could still be useful to be used via PuLP for solvers that don't have a direct PuLP backend

Python MIP (Mixed-Integer Linear Programming) Tools (new 2018)

PICOS - a user friendly Python API to several conic and integer programming solvers, very much like YALMIP or CVX under MATLAB.

C, C++ solver abstractions

or-tools (#33493) ​linear_solver wrapper

  • supporting CLP, CBC, GLPK, Gurobi, SCIP, XPRESS
  • permissive open source license: Apache 2.0
  • quick with "wontfix" - google/or-tools#3205

COIN-OR OSI

  • incompatible copyleft license: Eclipse Public License 2.0

Key Python software (solver-dependent)

The Python interfaces that target only one solver are usually careful in not having more restrictive licenses than the targeted solver.

scipy.optimize (vendors HiGHS)

  • Permissive open source licenses: SciPy: BSD 3-clause; HiGHS: MIT license
  • #32282 Add LP solver backends for HiGHS via scipy.optimize.linprog

highspy (pybind11-based interface to HiGHS)

  • #33919 Add MILP solver backends for HiGHS via highspy

Interfaces to GLPK

cylp

  • #33487: CyLP is a Python interface to COIN-OR’s Linear and mixed-integer program solvers (CLP, CBC, and CGL). CyLP’s unique feature is that you can use it to alter the solution process of the solvers from within Python. For example, you may define cut generators, branch-and-bound strategies, and primal/dual Simplex pivot rules completely in Python.
  • same license as CBC/CLP/CGL
  • as of Mar 2022: many open issues, minimal maintenance mode by 1 maintainer who is cooperative on receiving PRs: coin-or/CyLP#153, coin-or/CyLP#155, coin-or/CyLP#156, coin-or/CyLP#157

cbcpy (LGPL; new August 2019 according to ​https://pypi.org/project/cbcpy/; no activity since then)

Gurobi, CPLEX

  • they come with their own standard Python APIs, which we could use instead of building our own cython interface
  • related: #28175 - where we remove these cython modules from sagelib and ship them in separate packages instead

python-qsoptex

PySCIPOpt

  • #21003
  • PySCIPOpt: MIT license; SCIP: non-free "academic" license

Other relevant software

Possible integration routes via Julia

Change History (84)

comment:1 Changed 4 years ago by mkoeppe

  • Cc a.mason@… gh-jiawei-wang-ucd yzh dimpase dcoudert tmonteil added
  • Description modified (diff)
  • Summary changed from MIP frontend/backend using PuLP; backend to OSI using yaposib to Meta-ticket: Use Python optimization interfaces: PuLP, Pyomo, cylp...

comment:2 Changed 4 years ago by tmonteil

This was in my plans for sage days84 at Olot, but i did not go further since then. Also, there is Numberjack, which seems pretty general. Looking quickly at the repositories, it seems that neither Numberjack nor PuLP were modified recently (though we could contact the authors to see if they have plans for the future).

comment:3 Changed 4 years ago by dcoudert

I have the same feeling that PuLP has only minimal maintenance these days. Also, there is a significant effort in the OR community with Julia http://www.juliaopt.org/.

comment:4 Changed 4 years ago by mkoeppe

Andrew Mason commented by email:

PuLP is actively developed. It had a release last week. https://github.com/coin-or/pulp I know the developer Stuart Mitchell if you have any questions.

comment:5 Changed 4 years ago by mkoeppe

  • Description modified (diff)

comment:6 Changed 3 years ago by mkoeppe

  • Description modified (diff)

comment:7 Changed 3 years ago by mkoeppe

  • Description modified (diff)

Added Python-MIP

comment:8 Changed 3 years ago by mkoeppe

  • Description modified (diff)

comment:9 Changed 3 years ago by mkoeppe

  • Description modified (diff)

comment:10 follow-up: Changed 3 years ago by mkoeppe

https://pypi.org/project/mip/ looks good. Too bad we can't use it because its license, Eclipse Public License 2.0, is used without a Secondary Licenses Notice that would make it instantly GPL-compatible.

comment:11 in reply to: ↑ 10 ; follow-up: Changed 3 years ago by dimpase

Replying to mkoeppe:

https://pypi.org/project/mip/ looks good. Too bad we can't use it because its license, Eclipse Public License 2.0, is used without a Secondary Licenses Notice that would make it instantly GPL-compatible.

it's py3 only, by the way.

W.r.t. licenses, maybe we can ask the authors for a fix?

comment:12 Changed 3 years ago by mkoeppe

  • Cc fbissey added
  • Description modified (diff)

Added info on cbcpy and glpk from fbissey in #28175.

comment:13 in reply to: ↑ 11 Changed 3 years ago by mkoeppe

Replying to dimpase:

Replying to mkoeppe:

https://pypi.org/project/mip/ looks good. Too bad we can't use it because its license, Eclipse Public License 2.0, is used without a Secondary Licenses Notice that would make it instantly GPL-compatible.

it's py3 only, by the way.

W.r.t. licenses, maybe we can ask the authors for a fix?

Good idea... please go ahead.

comment:14 Changed 3 years ago by mkoeppe

  • Description modified (diff)

comment:15 Changed 3 years ago by mkoeppe

  • Description modified (diff)

comment:16 Changed 3 years ago by mkoeppe

  • Description modified (diff)

comment:17 Changed 2 years ago by mkoeppe

  • Milestone changed from sage-wishlist to sage-9.2

Moving some tickets to 9.2. This is not a promise that I will be working on them.

comment:18 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:19 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:20 Changed 2 years ago by mkoeppe

  • Milestone changed from sage-9.2 to sage-9.3

comment:21 Changed 19 months ago by mkoeppe

  • Description modified (diff)

comment:22 Changed 17 months ago by mkoeppe

  • Milestone changed from sage-9.3 to sage-9.4

Sage development has entered the release candidate phase for 9.3. Setting a new milestone for this ticket based on a cursory review of ticket status, priority, and last modification date.

comment:23 Changed 14 months ago by mkoeppe

  • Description modified (diff)

comment:24 Changed 13 months ago by mkoeppe

  • Milestone changed from sage-9.4 to sage-9.5

comment:25 Changed 13 months ago by mkoeppe

  • Description modified (diff)

comment:26 Changed 13 months ago by mkoeppe

  • Description modified (diff)

comment:27 Changed 13 months ago by gh-sheerluck

Pyomo can use SHOT sover https://github.com/coin-or/SHOT

When SHOT is configured to use all cores of cpu, solving traveling salesman problem is pretty fast.

comment:28 Changed 9 months ago by gh-sheerluck

In Python3.10 they removed use_tracing and I can not build https://github.com/coin-or/CyLP anymore :(

comment:29 Changed 9 months ago by mkoeppe

Reported upstream?

comment:31 Changed 8 months ago by mkoeppe

  • Milestone changed from sage-9.5 to sage-9.6

comment:32 Changed 5 months ago by mkoeppe

  • Milestone changed from sage-9.6 to sage-9.7

comment:33 Changed 5 months ago by gh-sheerluck

if installing CyLP for CBC is cumbersome, one can use https://github.com/coin-or/python-mip for CBC (https://docs.python-mip.com/en/latest/examples.html)

comment:34 Changed 5 months ago by mkoeppe

Unfortunately python-mip is encumbered with the same incompatible license as CBC itself. My plan is to target cvxpy as our main solver interface. It's well-maintained and supports lots of solver backends.

comment:35 follow-ups: Changed 5 months ago by gh-tkralphs

Hi all, I had lost track of this discussion after the IMA workshop and am now re-engaging. @mkoeppe, I looked back at some e-mail correspondence on this topic, but didn't really find a substantive answer there as to why license compatibility is an issue here. I understand that this is something probably prescribed by the Sage organization itself and this may not be the forum for this discussion. If not, I'd be happy to engage somewhere else.

It looks to me like this is a pretty conservative interpretation. It is certainly true that you cannot distribute a single binary that mixes GPL'd and EPL'd code, or a binary that calls a shared library where the two have different licenses (although one might claim it's OK if the main binary is GPL and it's the library that is EPL). But something like python-mip is a self-contained program in the Python programming language and it's not being distributed in binary form, but source form. Yes, when it uses Cbc, it calls a shared library that is under the EPL, but since python-mip itself is also under the EPL, this doesn't create any issue. I can't really see how re-distributing python-mip is a problem, regardless of what it is being re-distributed with. It is well-known that "EPL and GPL are incomatible," but I think this incompatibility is referring to a completely different context than we have here. Can someone say exactly which license is being violated and exactly how the violation is occurring?

PuLP is even more clear cut, since it is in pure Python and only needs a self-contained Cbc binary to work. It is certainly no problem to re-distribute a statically linked Cbc binary in combination with anything. It doesn't link to anything and is itself a completely self-contained program, which can be freely re-distributed. We wouldn't claim PuLP cannot be distributed because it has the capability to use CPLEX. Further, PuLP is itself under a compatible license (MIT?).

Cvxpy may have a compatible license, but it's not useful on its own for solving LPs/MILPs without installing some solver and the open-source solver it uses for solving MILPs is Cbc through CyLP. Effectively, this just puts the onus on the user, then, to install CyLP. This doesn't really seem to be a solution.

In any case, wouldn't the legal liability derive from an owner of a GPL'd code complaining that said code was mixed illegally with non-GPL'd code? I guess that Sage itself is the entity with the GPL'd code and I would think that this could simply be allowed. There is no issue from the EPL side and in any case, the risk of legal action stemming from the re-distribution of COIN-OR code is extremely small.

Why is Sage different than, e.g., anaconda or even SciPy?? Anaconda seems to mix all kinds of licenses and as far as I can tell, the onus is on the user to makes sure that the combination of packages installed on their own machine is OK. Anaconda can really be interpreted as an installer of other people's packages. But I didn't research this extensively or dig into whether anaconda actually re-distributes or just installs packages pulled from Pypi and other sources. Another data point would be Linux distros, which obviously do redistribute combinations of self-contained software packages under many different licenses, including both GPL and EPL.

It would be a shame if this license stuff gets in the way here when I think it is clear that all the players involved would approve of what is being done.

Last edited 5 months ago by gh-tkralphs (previous) (diff)

comment:36 in reply to: ↑ 35 Changed 5 months ago by mkoeppe

Hi Ted, thanks for joining the discussion here!

Replying to gh-tkralphs:

Why is Sage different than, e.g., anaconda [.... or ....] Linux distros, which obviously do redistribute combinations of self-contained software packages under many different licenses, including both GPL and EPL.

Yes, this would be "mere aggregation" https://www.gnu.org/licenses/old-licenses/gpl-2.0-faq.en.html#MereAggregation

comment:37 in reply to: ↑ 35 Changed 5 months ago by mkoeppe

Replying to gh-tkralphs:

Why is Sage different than, e.g., anaconda or even SciPy??

No difference to SciPy? really. SciPy? vendors a lot of packages, but all of them use a permissive license that is compatible with SciPy?'s own license, BSD 3-clause (https://github.com/scipy/scipy/blob/main/LICENSE.txt). It certainly does not vendor libraries with copyleft licenses such as GPL or EPL.

comment:38 in reply to: ↑ 35 Changed 5 months ago by mkoeppe

Replying to gh-tkralphs:

something like python-mip is a self-contained program in the Python programming language and it's not being distributed in binary form, but source form. Yes, when it uses Cbc, it calls a shared library that is under the EPL, but since python-mip itself is also under the EPL, this doesn't create any issue.

It would not be a problem for Sage if we just included python-mip as a package in the Sage distribution, for users to install and use.

But if we were to make substantial use of it in the Sage library, replacing our home-grown (wasn't me!) MIP interface (https://github.com/sagemath/sage/blob/develop/src/sage/numerical/mip.pyx), then this use would definitely rise above "mere aggregation" of software, making the result non-distributable in binary form.

Last edited 5 months ago by mkoeppe (previous) (diff)

comment:39 in reply to: ↑ 35 Changed 5 months ago by mkoeppe

Replying to gh-tkralphs:

Yes, when it uses Cbc, it calls a shared library that is under the EPL, but since python-mip itself is also under the EPL, this doesn't create any issue.

Yes, there is no new problem of using python-mip as an interface to Cbc when one is already using Cbc.

However, as I wrote in our earlier discussion (with Haroldo and Túlio) in July 2021:

For Sage, I am shopping for an API offering pivot-level control over solvers that can support several backends, with the goal of replacing our home-grown (wasn't me) design. Given that Python-MIP already supports CBC and Gurobi, it is a viable candidate for this; if it weren't for the license situation [...]

comment:40 in reply to: ↑ 35 Changed 5 months ago by mkoeppe

Replying to gh-tkralphs:

PuLP is even more clear cut, since it is in pure Python and only needs a self-contained Cbc binary to work. It is certainly no problem to re-distribute a statically linked Cbc binary in combination with anything. It doesn't link to anything and is itself a completely self-contained program, which can be freely re-distributed. We wouldn't claim PuLP cannot be distributed because it has the capability to use CPLEX. Further, PuLP is itself under a compatible license (MIT?).

Yes, permissive licenses such as the one that PuLP uses (and the one that python-mip used to use...) do not create such compatibility problems. It is unproblematic to distribute PuLP with Cbc in binary form. (Obviously, you can't redistribute the combination of PuLP with CPLEX.)

It is specifically the switch of python-mip from a permissive license to the incompatible copyleft license (EPL without "secondary license notice") that created the incompatibility with GPL software.

Last edited 5 months ago by mkoeppe (previous) (diff)

comment:41 in reply to: ↑ 35 Changed 5 months ago by mkoeppe

Replying to gh-tkralphs:

Cvxpy may have a compatible license, but it's not useful on its own for solving LPs/MILPs without installing some solver and the open-source solver it uses for solving MILPs is Cbc through CyLP. Effectively, this just puts the onus on the user, then, to install CyLP. This doesn't really seem to be a solution.

That's not a problem for the Sage distribution, we have a working packaging infrastructure that already vendors hundreds of packages - https://repology.org/projects/?inrepo=sagemath_develop

comment:42 follow-ups: Changed 5 months ago by gh-tkralphs

OK, thanks for clarifying, but it still looks to me like the GPL actually doesn't restrict things in the way you (and many others) are interpreting. Every argument I find that claims Python modules under incompatible licenses cannot be combined in this way takes as given that "importing a module" in Python is equivalent to "linking to a library" in a compiled language and this just isn't true in any sense. "Importing" in Python is functionally the same as "including" a source file in C/C++. You are allowed to combine GPL'd source code with other source code, no matter what the license and re-distribute the combination in source form, just not in binary form.

If you think about the philosophical underpinnings, what the GPL tries to prevent is that I modify the source code of a GPL'd code (or something linked to a GPL'd code) and then re-distribute it only in binary form, thereby profiting from a work that is derived from a GPL'd work, but not sharing the source of that derived work. The viral nature is just meant to try to ensure that everyone plays by these rules to the extent possible by preventing anyone from linking to GPL'd code in binary form without playing by the rules. But unless you distribute only a .pyc file and not a .py, this simply can't happen in Python. Although it would be possible to distribute a Python package in byte-compiled form without source (or even in some obfuscated form of source), this is not what you would be planning to do and it just doesn't run afoul of the GPL, either by the letter of the law or in spirit.

Of course, I am not a lawyer, but in my opinion, you can take python-mip, modify it as you wish, and re-distribute the modified version as an integrated part of Sage without any license issues. Here is an answer on Open Source Stack Exchange that I think is correct and supports this point of view, although everyone else seems to miss the point and disagree.

https://opensource.stackexchange.com/a/8472

Last edited 5 months ago by gh-tkralphs (previous) (diff)

comment:43 in reply to: ↑ 42 Changed 5 months ago by mkoeppe

I'd say this narrow view on what "linking" is amounts to wishful thinking. It's pretty clear from https://www.gnu.org/licenses/old-licenses/gpl-2.0-faq.en.html#TOCMereAggregation that the intention of the license is broader.

Last edited 5 months ago by mkoeppe (previous) (diff)

comment:44 Changed 5 months ago by mkoeppe

More importantly, as an author of GPL-protected Python software myself, I would certainly not want this interpretation to be used when a vendor of proprietary software wanted to build their product on top of my software.

comment:45 Changed 5 months ago by gh-sheerluck

Thanks to both of you for detailed explanation. I was trying to use cvxpy with CBC without CyLP and ended up injecting code from https://github.com/jurasofish/mip_cvxpy/blob/develop/mip_cvxpy/__init__.py inside https://github.com/cvxpy/cvxpy/blob/master/cvxpy/reductions/solvers/conic_solvers/cbc_conif.py

comment:46 Changed 5 months ago by mkoeppe

  • Description modified (diff)

I've opened #33487 for adding a package cylp

comment:47 in reply to: ↑ 42 Changed 5 months ago by mkoeppe

A last, minor point on this:

Replying to gh-tkralphs:

unless you distribute only a .pyc file and not a .py, this simply can't happen in Python. Although it would be possible to distribute a Python package in byte-compiled form without source (or even in some obfuscated form of source)

Even for pure Python packages and even without intentional obfuscation, the Python files installed in site-packages would not be "source code" in the sense of the license, which is the "preferred form of the work for making modifications to it": The build infrastructure (setup.py, pyproject.toml) is not installed.

comment:48 Changed 5 months ago by mkoeppe

  • Description modified (diff)

comment:49 follow-ups: Changed 5 months ago by gh-tkralphs

Sorry, but I can't help ranting a bit more about the GPL :-).

the intention of the license is broader

Sure, maybe, but that's not how it works. You can't take a vague and (what looks to me to be) unenforceable license and paper over it with FAQs that fill in the gaps ex post facto. The FSF consistently and loudly promotes interpretations of the GPL that seem to me to also be "wishful thinking." Intentions are not really relevant, what matters is how a court would ultimately interpret the license from a legal standpoint and what provisions of it the court would determine are actually enforceable. There are certainly plenty of people who find the FSF's claims about enforceability of certain provisions of the GPL to be pretty questionable. The license text itself is vague and its interpretation by the FSF is pretty over-reaching. It's very difficult to say one way or the other on a wide range of subtle questions around scenarios the license doesn't seem to have been designed to take into account, such as interpreted languages.

A lot of what is explained in various places about how the GPL "should" be interpreted isn't actually in the license and requires a pretty imaginative reading of the actual verbiage. But it is all a matter of opinion and the only opinion that really matters is judge and jury.

Even for pure Python packages and even without intentional obfuscation, the Python files installed in site-packages would not be "source code" in the sense of the license, which is the "preferred form of the work for making modifications to it": The build infrastructure (setup.py, pyproject.toml) is not installed.

This is a very subtle distinction that is one I really doubt would stand up to legal scrutiny. There are many subtleties around this interpretation. For one, does this mean that you can't actually grab part of a package and utilize it in another package in source form, since the partial package would not be in the "preferred form for modification"? This seems to be done quite commonly. If not, what actually defines what constitutes the "full package"? What about a case like COIN-OR, where you can't work with the build system of any of the main source package without what is clearly another completely separate package that aids in the production of the build harness (in this case, the BuildTools?)? What if the original authors don't actually provide a build harness, but just distribute the source itself? What if the project does have a setup.py, but can be run without it, by just directly importing it, not installing it in site-packages? Wouldn't this then be the most convenient way of working with the code? In the C/C++ world, what if the source is provided, but without a Makefile or a CMake setup? Or what if both setups are maintained for a given project, but only one is distributed and it's not the one I would personally prefer? Who's to say what the "preferred" way of working with source is, given that this is subjective. This is a crazy mess. To me, it seems clear that the only distinction one can make very clearly is that "source" is human-readable and "object code" is not.

Putting all that aside, a Python wheel certainly can include things like setup.py if desired and those things would then be installed. Doing that would certainly seem to satisfy license requirements. There's nothing stopping you from including the entire contents of a given project's repo in its wheel, it's just not usually done that way.

There are many other aspects of the GPL that are up to interpretation and with respect to which it is anyone's guess what is really allowed. If you look at the parts of the license itself (and not the FAQ) that discuss linking, it only references linking to libraries that are "required." The word "required" seems to be intentionally chosen there. It looks as though if you distribute an optional component along with the whole work, that optional component doesn't itself need to be under the GPL. This makes sense---an optional plug-in cannot really be interpreted as part of a whole that is a derived work of the original. The argument about the enforceability of "linking" rests on the idea that the combination of the two things linked together creates something new that would be considered a derived work. Since python-mip is itself a free-standing program that works without Sage and Sage is a free-standing program that works without python-mip, this really doesn't look like "linking" in that sense. The only place that "linking" is really defined in the license itself, it is specifically mentioned that the linking is to "shared libraries and dynamically linked subprograms," which is clearly referring to code in assembly code form, not a human-readable source form.

Of course, this is all opinion, but it seems clear there is enough grey to cast doubt on almost any one fixed interpretation. Even if you take the conservative view that importing a module in Python is "linking," it would seem easy to get around this by invoking the module as a free-standing program, writing out the data to a file and then reading it back in using a separate interpreter with its own memory. Since we agree that passing data through a file rather than through memory is a legitimate way to keep the programs as separate (ala how PuLP communicates with Cbc), this is a fine workaround.

But then this opens up a whole new subtle distinction between the RAM and the disk. Is the disk not part of the computer's "memory"? It seems like it is, since when a program pages, it is using the disk as virtual memory. What then is considered communicating through "memory"? What about RAM on another processor or attached to another core? Is that "memory"? Where do we draw the line? It's almost impossible to clearly define exactly what the GPL requires in today's complex computing environments. The FSF is splitting hairs in its interpretation. It's hard to believe that they are truly trying to respect a user's "freedom" with all this. It looks to me like they are trying to forcibly impose an extreme world view on everyone else, thereby taking away freedom, to the detriment of the open source movement.

With all that said, the most important point at the end of the day is that using python-mip in the way you were hoping to use it does not violate the EPL. It only (possibly) violates the GPL. I guess the Sage foundation is the copyright holder on the GPL'd code and so could carve out an exception to the linking requirement for specific purposes. This is effectively what the secondary clause of the EPL also does. Since it is the GPL, not the EPL, that is creating the problem, why not just address it on the GPL side. Why not use the LGPL?

More importantly, as an author of GPL-protected Python software myself, I would certainly not want this interpretation to be used when a vendor of proprietary software wanted to build their product on top of my software.

How does the linking provision provide protection? It just means that no one can build on top of your software with non-GPL'd code, not that they can't do it at all. Lots of commercial products are built on GPL'd code. The protection the GPL affords you is that if someone modifies your code, they have to make the modifications open source and they can't hide from users the fact that their project is built on yours. But they can still build a commercial product around it and charge money for it.

comment:50 in reply to: ↑ 49 Changed 5 months ago by mkoeppe

Replying to gh-tkralphs:

Since we agree that passing data through a file rather than through memory is a legitimate way to keep the programs as separate (ala how PuLP communicates with Cbc)

No, we don't agree on that.

comment:51 in reply to: ↑ 49 Changed 5 months ago by mkoeppe

Replying to gh-tkralphs:

there is enough grey to cast doubt on almost any one fixed interpretation.

I agree with you on that --- which is why I will avoid building software on top of brittle interpretations such as the one that you are trying to give in order to arrive at the desired result.

comment:52 in reply to: ↑ 49 Changed 5 months ago by mkoeppe

Replying to gh-tkralphs:

thereby taking away freedom

That's what COIN-OR did to users of python-mip by imposing the copyleft EPL license on it, replacing the previous permissive license -- which is the actual problem that I pointed out in comment:34 (and our email exchange in 2021).

Last edited 5 months ago by mkoeppe (previous) (diff)

comment:53 in reply to: ↑ 49 Changed 5 months ago by mkoeppe

Replying to gh-tkralphs:

Why not use the LGPL?

In the case of Sage, obviously because of Wolfram, MapleSoft, and MathWorks.

Last edited 5 months ago by mkoeppe (previous) (diff)

comment:54 in reply to: ↑ 49 Changed 5 months ago by mkoeppe

Replying to gh-tkralphs:

the intention of the license is broader

Sure, maybe, but that's not how it works. [...] But it is all a matter of opinion and the only opinion that really matters is judge and jury.

Sorry, Ted, this is all just wildly misguided.

The authors of software choose GPL as the license because they agree with the widely understood intended interpretation, in particular the strong protections that the GPL intends to provide.

Whether the implementation by the license text holds up legally, is a secondary matter; the license is, so to say, merely an implementation detail.

What matters for the open source / free software community is to respect the intent of the authors, as expressed by their choice of license. Saying that their choices are somehow not legitimate because the license is not legally enforceable, or to discuss circumvention techniques based on narrow interpretations of what linking is, all seems pretty out of place.

comment:55 in reply to: ↑ 49 Changed 5 months ago by dimpase

Replying to gh-tkralphs:

Sorry, but I can't help ranting a bit more about the GPL :-).

the intention of the license is broader

Sure, maybe, but that's not how it works. You can't take a vague and (what looks to me to be) unenforceable license

It's sad to see how well shameless anti-GPL propaganda works. GPL violations have been successfully enforced by US courts - as copyright violations (cf. Jacobsen v. Katzer), and even as contract violations (Artifex v. Hancom).

comment:56 follow-ups: Changed 5 months ago by gh-tkralphs

I am sorry, I didn't intend to offend anyone or spout what sounded like propaganda in a public forum. Since I have known Matthias IRL for many years and have deep respect for him and his work, I let my guard down a bit and probably wasn't careful enough in what I said. I can see how some of what I said could be interpreted as misguided and I apologize for that. However, there is a discussion worth having here if cooler heads can prevail and I think some of what I said was misconstrued. I am open to listening and moderating my point of view and would appreciate the chance to further engage, so consider this an open invitation to have a more in-depth discussion in another appropriate forum. I believe we are all reasonable people who all have valid points of view and can have a rational discourse on this complicated, nuanced topic.

I certainly didn't mean to promote undermining or circumventing the intents of individual rights-holders, which should be respected, and I also didn't mean to denigrate anyone's individual choices or characterize them as illegitimate. I'm a passionate open source developer myself and have tried to do my own independent thinking about these issues over years of working with open source licenses. Yes, my views are biased by the frustration of seeing potential projects that seem to be an obvious "good thing" for the world and that I can't see as objectionable to any of the individual rights-holder stopped in their tracks by this license madness. I am frustrated by the divisiveness and the political agendas. I don't assume the FSF speaks for all those who adopt the GPL and would prefer to directly engage with the rights-holders of any code I intend to work with to understand their point of view and intents.

As a devoted open source developer over more than two decades, my only agenda is enabling people to benefit from the free software I'm contributing to and to grow the community of engaged developers. It would be a win for COIN-OR to be able to partner with Sage. So naturally, I'm looking for a way through this license maze that will allow us to do good work together, of course without undermining the intents of any fellow open source developers. This particular scenario in which the goal is to use an EPL'd code to provide additional functionality within an overall GPL'd code seems harmless to me. The prohibition of this by the linking restriction seems like collateral damage more than something that it by design.

That's what COIN-OR did to users of python-mip by imposing the copyleft EPL license on it, replacing the previous permissive license -- which is the actual problem that I pointed out in comment:34 (and our email exchange in 2021).

I did want to clarify that COIN-OR projects can use any OSI-approved license, including the GPL. Nothing has been imposed. Most have chosen the EPL, following IBM's original lead. Of course, this is partly because it is just easier in terms of license compatibility, but forcing the EPL on people is not the goal, it is just unfortunate side effect that looks to me like it derives from the GPL's prohibition against mixing licenses. The GPL and the EPL actually seem pretty similar in broad philosophy, the big difference being that the EPL is less restrictive on mixing packages under different licenses (yes, I know there are other differences that people view as important).

I will avoid building software on top of brittle interpretations such as the one that you are trying to give in order to arrive at the desired result.

This encapsulates my frustration. Because of the threat of legal action by the FSF, developers with good intentions choose to avoid doing things that actually seem to make sense and that (I don't think) violate anyone else's intent. The FSF seems to want to strong-arm everyone into switching to the GPL. This just makes me bristle. I understand that it is exactly the threat of legal action that gives the GPL its strong protection, but I think there are better ways of going about this that would not be so divisive or result in so much collateral damage.

There's a lot more to be said, but this is obviously not the right place to do it. The invitation to further discussion is open. I hope someone takes me up on it.

Version 0, edited 5 months ago by gh-tkralphs (next)

comment:57 in reply to: ↑ 56 Changed 5 months ago by mkoeppe

Replying to gh-tkralphs:

I did want to clarify that COIN-OR projects can use any OSI-approved license, including the GPL. Nothing has been imposed.

But COIN-OR's embrace of python-mip has prompted the license change.

The old license was a standard permissive license that was compatible with everything: with EPL, with GPL, anything.

Most have chosen the EPL, following IBM's original lead. Of course, this is partly because it is just easier in terms of license compatibility

Yeah, no. This is the core of the damaging misinformation that is coming from the direction of COIN-OR.

unfortunate side effect that looks to me like it derives from the GPL's prohibition against mixing licenses.

No such thing, it is just specifically incompatible with EPL (without secondary license notice), for a specific technical reason.

GPL is compatible with a wide range of licenses, in particular all the standard permissive licenses (MIT, BSD, Apache, ...).

comment:58 in reply to: ↑ 56 Changed 5 months ago by mkoeppe

Replying to gh-tkralphs:

I don't assume the FSF speaks for all those who adopt the GPL and would prefer to directly engage with the rights-holders of any code I intend to work with

Yes, it makes no sense to bring up the FSF at all. They published the license; they are not the copyright holder in any software that we are talking about.

comment:59 in reply to: ↑ 56 Changed 5 months ago by mkoeppe

Replying to gh-tkralphs:

This particular scenario in which the goal is to use an EPL'd code to provide additional functionality within an overall GPL'd code seems harmless to me.

Yes, which is why we've had no problem in packaging CBC as an optional package that replaces the default solver backend with a better one. And there's also no problem in doing the same with CyLP (#33487).

The situation with python-mip is different, as I explained in comment:38: I was looking into adopting its API as the basis for our interface development, with the goal of using it for several backends. This intended closer integration is no-go with the incompatible license, so it is not going to happen.

comment:60 Changed 5 months ago by mkoeppe

Ted, I of course appreciate all the hard work and dedication that you have put in maintaining COIN-OR, which among other things preserves CBC, an important open source jewel. CBC/CLP/...'s license incompatibility with the wide range of GPL software is a sad fact regarding this legacy code, for which there is unfortunately no solution.

But COIN-OR would be well-advised to stop spreading this problem to new code.

comment:61 Changed 5 months ago by mkoeppe

  • Description modified (diff)

comment:62 Changed 5 months ago by mkoeppe

  • Description modified (diff)

comment:63 Changed 5 months ago by mkoeppe

  • Description modified (diff)

comment:64 Changed 5 months ago by mkoeppe

  • Description modified (diff)

comment:65 Changed 5 months ago by mkoeppe

  • Summary changed from Meta-ticket: Use Python optimization interfaces: PuLP, Pyomo, cylp... to Meta-ticket: Use Python optimization interfaces: CVXPY, or-tools, PuLP, Pyomo, cylp...

comment:66 Changed 5 months ago by mkoeppe

  • Description modified (diff)

comment:67 Changed 5 months ago by mkoeppe

  • Description modified (diff)

comment:68 Changed 5 months ago by mkoeppe

  • Description modified (diff)

comment:69 Changed 5 months ago by mkoeppe

  • Description modified (diff)

comment:70 Changed 5 months ago by mkoeppe

  • Description modified (diff)

comment:71 Changed 5 months ago by mkoeppe

  • Description modified (diff)

comment:72 Changed 5 months ago by mkoeppe

  • Description modified (diff)

comment:73 in reply to: ↑ 56 Changed 5 months ago by mkoeppe

Replying to gh-tkralphs:

I am sorry, I didn't intend to offend anyone

No worries, you did mark it as <rant>, but the end tag is missing

comment:74 Changed 5 months ago by mkoeppe

  • Description modified (diff)

comment:75 Changed 5 months ago by mkoeppe

  • Description modified (diff)

comment:76 Changed 5 months ago by mkoeppe

  • Description modified (diff)

comment:77 Changed 5 months ago by mkoeppe

  • Description modified (diff)

comment:78 Changed 5 months ago by mkoeppe

  • Description modified (diff)

comment:79 Changed 5 months ago by mkoeppe

  • Description modified (diff)

comment:80 Changed 5 months ago by mkoeppe

  • Description modified (diff)

comment:81 Changed 3 months ago by mkoeppe

  • Description modified (diff)

comment:82 Changed 3 months ago by mkoeppe

  • Milestone changed from sage-9.7 to sage-9.8

comment:83 Changed 3 months ago by mkoeppe

  • Description modified (diff)

comment:84 Changed 2 hours ago by mkoeppe

  • Description modified (diff)
Note: See TracTickets for help on using tickets.