Opened 14 months ago

Last modified 8 weeks ago

#28434 new defect

Syntax for callable symbolic expressions causes too much confusion

Reported by: rburing Owned by:
Priority: major Milestone: sage-9.3
Component: symbolics Keywords: CallableSymbolicExpression, function, callable
Cc: Merged in:
Authors: Reviewers:
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Description (last modified by rburing)

The syntax for callable symbolic expressions works in the simplest cases, but beyond that it breaks down and causes much confusion, as can be often seen on Ask SageMath and (in my experience) when trying to teach SageMath to people. Here is a non-exhaustive list of 8 examples:

  1. Numbers confusion:
sage: f(x) = x^2
sage: f(2).factor()
4
  1. Polynomial confusion:
sage: R.<z,w> = PolynomialRing(QQ)
sage: f(x) = x^2
sage: f(z+w).coefficient({z : 1})
...
TypeError: no canonical coercion from <type 'dict'> to Symbolic Ring

https://ask.sagemath.org/question/47064/how-to-turn-the-function-into-expression/

  1. Matrix confusion:
sage: B(x) = matrix([[x, 0], [0, 0]])
sage: B(12)
[x 0]
[0 0]

https://ask.sagemath.org/question/10457/arithmetic-with-matrices-of-formal-functions/

  1. List confusion:
sage: f(x) = [x,x]
sage: f(2).parent()
Vector space of dimension 2 over Symbolic Ring

https://ask.sagemath.org/question/10449/how-to-return-a-list-from-callable-symbolic-expression/

  1. Derivative confusion (and argument confusion):
sage: f(x) = x.derivative()
sage: f(x^2)
1
sage: f(y) = y.derivative(x)
sage: f(x^2)
0

https://ask.sagemath.org/question/9842/the-difference-between-fx3-and-f3-of-callable-symbolic-expression-f/

  1. Matrix argument confusion:
sage: f(x) = x^2
sage: f(2*identity_matrix(2))
...
TypeError: no canonical coercion from Full MatrixSpace of 2 by 2 dense matrices over Integer Ring to Callable function ring with argument x

https://ask.sagemath.org/question/38524/defining-functions-acting-on-matrix-elements/

  1. Addition confusion:
sage: f(x) = x^2
sage: g(x) = x^2
sage: var('t')
sage: h(t) = t^2
sage: f+g
x |--> 2*x^2
sage: f+h
(t, x) |--> t^2 + x^2

https://ask.sagemath.org/question/10782/symbolic-functions-without-named-variables/

  1. Non-symbolic function confusion:
sage: tau(n) = len(divisors(n))
...
TypeError: 'sage.symbolic.expression.Expression' object is not iterable

https://ask.sagemath.org/question/47672/is-this-a-bug-or-intended-behavior/

Change History (11)

comment:1 Changed 14 months ago by rburing

  • Description modified (diff)

comment:2 follow-up: Changed 14 months ago by jdemeyer

Let's look at these one by one:

  1. The most obvious bug IMHO. Factoring in SR, especially in obvious cases like this, should work. This has nothing to do with callable expressions though.
  2. Not really a bug, you shouldn't mix symbolic expressions and polynomials.
  3. Bug. Substituting in matrices should work.
  4. Not a bug, you cannot make a list symbolic.
  5. Not a bug, the RHS x.derivative() equals 1 so f(x) = x.derivative() is just a complicated way to write f(x) = 1.
  6. Bug, there is no reason why this shouldn't work.
  7. I don't see the problem here, what would you expect?
  8. Not a bug, similar to 5: the RHS divisors(n) is meaningless for a symbolic n.

Some of these (in particular 4, 5, 8) are really user mistakes where an ordinary Python function is meant instead of a symbolic function. I agree that this may be confusing, but there really is an important difference between

f(x) = x.derivative()

and

def f(x):
    return x.derivative()

I recommend you to create separate tickets for the bugs that you want to see fixed. I cannot at all promise that they will be fixed. But it will help discussion if we don't have a single issue which is about multiple unrelated things.

comment:3 in reply to: ↑ 2 ; follow-up: Changed 14 months ago by rburing

I have to disagree that these issues are unrelated. The main source of confusion (in my opinion) is that this syntax is reserved for something very particular (callable symbolic expressions), while it looks like something much more general, like a lambda-style definition of a Python function.

  1. The most obvious bug IMHO. Factoring in SR, especially in obvious cases like this, should work. This has nothing to do with callable expressions though.

What I meant to show was: the (naive) expectation is that when you put in an integer, you get back an integer. This is false, because what you get is a symbolic expression. So a better example (with a method that isn't available on symbolic expressions) would be f(2).binary().

  1. Not really a bug, you shouldn't mix symbolic expressions and polynomials.

They can often be mixed. A "fix" here is to write R(f(z+w)).coefficient({z : 1}). My point is: how does the user know that f(x) = x^2 defines a callable symbolic expression which should get this special treatment (compared to a Python function)? Or, how should a user know not to use these in this context?

  1. Not a bug, you cannot make a list symbolic.

How does the user know that this is an issue? It's not an issue with Python functions...

  1. Not a bug, the RHS x.derivative() equals 1 so f(x) = x.derivative() is just a complicated way to write f(x) = 1.

I agree that this may be confusing, but there really is an important difference between

f(x) = x.derivative()

and

def f(x):
    return x.derivative()

We know this and we are used to it, but in my opinion it really makes no sense. I can kind of understand the desire to create a type of function object (which is what happens in the first case), e.g. for prettyprinting, but the fact that these two f's don't give the same output when evaluated is honestly ridiculous. How should a user know?

  1. I don't see the problem here, what would you expect?

Some would expect two scalar-valued functions of a scalar variable to always add up to a scalar-valued function of a scalar variable. But I will concede this is the least confusing one.

  1. Not a bug, similar to 5: the RHS divisors(n) is meaningless for a symbolic n.

How does the user know that they are defining a symbolic callable expression, and that they shouldn't?

All the questions were rhetorical. In my experience (and in the posted threads), the user doesn't know, because it isn't clear. Sage would be less confusing without this syntax (in its current form).

comment:4 in reply to: ↑ 3 Changed 14 months ago by defeo

All the questions were rhetorical. In my experience (and in the posted threads), the user doesn't know, because it isn't clear. Sage would be less confusing without this syntax (in its current form).

So, is this ticket suggesting to remove the syntax? If so, I'm not convinced a ticket is the right place to have such a discussion.

Do you realize how much user code it would break? Maybe you should lay out your plan more clearly on sage-devel.

comment:5 follow-ups: Changed 14 months ago by nbruin

Thanks for putting the list together. It's nice to have a record which issues (often?) arise. A lot of confusion happens because of the mathematics underneath. Confusion is a common state for students who are learning mathematics and I think it is to be expected that they will also encounter it while working with a computer algebra system of significant scope and complexity. It's nice to reduce confusion where possible, but I think it's unrealistic to expect we can do away with it.

Some of your items are indicated above as bugs and can probably be solved. Other items might be better to put in an FAQ document (or the documentation) for people to point to if questions arise. I don't know if these examples are best kept track of on a ticket. The ticket itself isn't really resolvable, but there are other administrative tracker tickets around.

Answer to "how does a user know"?

sage: f(x)=x^2
sage: type(f)
<type 'sage.symbolic.expression.Expression'>
sage: parent(f)
Callable function ring with argument x

Once a user run into issues like this, it's clear that they have to progress beyond the level of "beginner" and start a more "advanced" approach. When they have questions concerning this, they have a motivation to read an "advanced" tutorial that goes more into the nuts and bolts of things. After that tutorial they should be at a level where they can make sense of the reference guide. Perhaps the "advanced" tutorial already exists. Perhaps it needs to be written?

Note that callable symbolic expressions serve a very definite purpose and are therefore not going away. The preparse trick of defining them f(x)=... is magic but incredibly useful for giving compact illustrations in educational settings, and that syntax doesn't seem to be a source for confusion anyway (other than that people don't get much of a cue that this is not a python built-in feature -- but letting people look at preparse("...") quickly illustrates what's going on. Callable symbolic expressions are necessary to use positional call syntax for symbolically defined functions/expressions, which is very common in math text books.

A general comment: Building a computer algebra system on top of python is necessarily a compromise. This is true for other python-based solutions too, for instance pandas or matplotlib. In most of these cases, a Domain Specific Language would be much more elegant and unified. Although, beware what you ask for: in for instance Maple, they DO have a very clear symbolic expression model that is fundamental to everything. And they found it doesn't address all their needs and had to bolt on other solutions. It makes for a decidedly unpleasant system to program for. They ended up with their compromises in other places.

1) symbolic factor COULD try and use integer factorization on objects that look like an integer or a rational number. That would change the complexity of the routine significantly (but perhaps for SR we don't care), and of course it might make people wonder why 2*I doesn't get factored.

5) doesn't seem to have to do with callable expressions at all. The fact that "derivative" is allowed to try to do something without an argument is necessary because of univariate calculus

6) Bug maybe, but possibly a consequence of design that will be hard to fix: "symbolic matrices" aren't members of the symbolic ring. They are matrices *over* SR. So filling them into a function that is from and to SR is problematic.

comment:6 in reply to: ↑ 5 ; follow-up: Changed 14 months ago by kcrisman

Note that callable symbolic expressions serve a very definite purpose and are therefore not going away. The preparse trick of defining them f(x)=... is magic but incredibly useful for giving compact illustrations in educational settings, and that syntax doesn't seem to be a source for confusion anyway (other than that people don't get much of a cue that this is not a python built-in feature -- but letting people look at preparse("...") quickly illustrates what's going on.

Thanks for a very well-organized resume of the issues at stake, Nils.

comment:7 in reply to: ↑ 6 Changed 14 months ago by egourgoulhon

Replying to kcrisman:

Thanks for a very well-organized resume of the issues at stake, Nils.

+1

comment:8 in reply to: ↑ 5 Changed 14 months ago by rburing

Thanks, Nils.

I had a look at the implementation. A callable symbolic expression like f(x) = x^2 is mostly just an Expression whose parent is a CallableSymbolicExpressionRing; it keeps track of the list of (symbolic variable) arguments and implements _call_element (for substitution) and _repr_element (for prettyprinting). The notation f(x,y) = (x,y) defines a vector over a CallableSymbolicExpressionRing, and the special element class Vector_callable_symbolic_dense is used to fix the notation with a _repr_ method.

This is quite simple and powerful, and I now agree that the notation should be kept. It seems that the best resolution is to fix the bugs, improve the documentation, improve error handling, and extend the functionality to include a matrix-valued variant. See a detailed proposal below.

I found that the Guided Tour already has a page Some Common Issues with Functions which is a great start for clarifying some of the issues. This should be the page to point people to when they run into any of this.

Let me go over the list again:

1) If f(x) = x^2 then f(2) is not an Integer

The Common Issues page point 2 should explain more precisely what a callable symbolic expression is. In particular, at definition-time, the arguments are symbolic variables (and treated as such on the right-hand side); calling means substitution, and the result (in the scalar-valued case) is a symbolic expression. Also, the "vector-valued" (and possibly "matrix-valued", see 3 below) variant deserves a mention.

2) Polynomial confusion

Covered by the previous point. I would like to have the "solution" to this (explicit conversion) as an example somewhere, but since the Common Issues page is part of the Guided Tour it would be too early there.

3) Matrix confusion

Currently a bug. I believe matrices are not much different from vectors, and a matrix-valued variant could be implemented. It should return a matrix over a CallableSymbolicExpressionRing, and a special element class with _repr_ and _latex_ (like Vector_callable_symbolic_dense) should fix the notation.

4) List confusion

In my opinion the syntax with square brackets for vector-valued functions is an abuse of notation (which has caused confusion with lists); it should be f(x) = (x,x) or f(x) = vector([x,x]) (the latter currently doesn't work, by the way). This is handled in symbolic_expression.

5) Derivative confusion (and argument confusion)

Handled by point 1. In particular, the point is that the arguments in the right-hand side of the definition of a callable symbolic expression are symbolic variables, not arbitrary objects, and in particular they cannot be functions.

6) Matrix argument confusion

From the implementation we know that _call_element is a substitution of symbolic variables. So all arguments must be (coerced to) symbolic expressions. Since this is not possible for a matrix, we can give a much better error here.

7) Addition confusion

I can live with this. Maybe it should be mentioned in the documentation.

8) Non-symbolic function confusion

Can we improve the preparser here? Currently, it translates tau(n) = len(divisors(n)) into

__tmp__=var("n");
tau = symbolic_expression(len(divisors(n))).function(n)

But how about this instead?

__tmp__=var("n");
try:
    tau = symbolic_expression(len(divisors(n))).function(n)
except Exception, e:
    raise TypeError, "Failed to define callable symbolic expression. Is the right-hand side a valid (tuple/vector/matrix of) symbolic expression(s) in symbolic variable(s) " + str(__tmp__) + "? Original exception: " + str(e)

The best part about this is that it informs the user that it is attempting to define a callable symbolic expression. If this is not what they meant, the user will probably do a web search for "Sage function" and will land on the (highly ranked) Common Issues page, which starts with the definition of ordinary Python functions.

comment:9 Changed 10 months ago by embray

  • Milestone changed from sage-8.9 to sage-9.1

Ticket retargeted after milestone closed

comment:10 Changed 6 months ago by mkoeppe

  • Milestone changed from sage-9.1 to sage-9.2

Batch modifying tickets that will likely not be ready for 9.1, based on a review of the ticket title, branch/review status, and last modification date.

comment:11 Changed 8 weeks ago by mkoeppe

  • Milestone changed from sage-9.2 to sage-9.3
Note: See TracTickets for help on using tickets.