Opened 3 years ago

Meta-ticket: Callable symbolic expressions

Reported by: Owned by: rburing major sage-9.9 symbolics CallableSymbolicExpression, function, callable slelievre, mjo, egourgoulhon N/A

GitHub link to the corresponding issue

The syntax for callable symbolic expressions works in the simplest cases, but beyond that it breaks down and causes much confusion, as can be often seen on Ask SageMath and (in my experience) when trying to teach SageMath to people. Here is a non-exhaustive list of 8 examples:

1. Numbers confusion:
```sage: f(x) = x^2
sage: f(2).factor()
4
```
1. Polynomial confusion:
```sage: R.<z,w> = PolynomialRing(QQ)
sage: f(x) = x^2
sage: f(z+w).coefficient({z : 1})
...
TypeError: no canonical coercion from <type 'dict'> to Symbolic Ring
```
1. Matrix confusion:
```sage: B(x) = matrix([[x, 0], [0, 0]])
sage: B(12)
[x 0]
[0 0]
```
1. List confusion:
```sage: f(x) = [x,x]
sage: f(2).parent()
Vector space of dimension 2 over Symbolic Ring
```
1. Derivative confusion (and argument confusion):
```sage: f(x) = x.derivative()
sage: f(x^2)
1
sage: f(y) = y.derivative(x)
sage: f(x^2)
0
```
1. Matrix argument confusion:
```sage: f(x) = x^2
sage: f(2*identity_matrix(2))
...
TypeError: no canonical coercion from Full MatrixSpace of 2 by 2 dense matrices over Integer Ring to Callable function ring with argument x
```
```sage: f(x) = x^2
sage: g(x) = x^2
sage: var('t')
sage: h(t) = t^2
sage: f+g
x |--> 2*x^2
sage: f+h
(t, x) |--> t^2 + x^2
```
1. Non-symbolic function confusion:
```sage: tau(n) = len(divisors(n))
...
TypeError: 'sage.symbolic.expression.Expression' object is not iterable
```
1. Derivative confusion.
```sage: f(t) = (t, t^2, t^3)

sage: f
t |--> (t, t^2, t^3)
sage: f(t)
(t, t^2, t^3)

sage: f.parent()
Vector space of dimension 3 over Callable function ring with argument t
sage: f(t).parent()
Vector space of dimension 3 over Symbolic Ring

sage: f1 = f(t).derivative(t)
sage: f1
(1, 2*t, 3*t^2)
sage: f1.parent()
Vector space of dimension 3 over Symbolic Ring

sage: g(t) = f(t).derivative(t)
Traceback (most recent call last)
...
TypeError: unable to convert (1, 2*t, 3*t^2) to a symbolic expression

sage: h = f.derivative()
sage: h
[    t |--> 1]
[  t |--> 2*t]
[t |--> 3*t^2]
sage: h.parent()
Full MatrixSpace of 3 by 1 dense matrices
over Callable function ring with argument t
```
1. Newline confusion. See #11621, #19088, #30953.
```sage: f(t) = (t,
....:         t^2,
....:         t^3)
File "<ipython-input-46-d381d3086d21>", line 2
t**Integer(2),
^
SyntaxError: invalid syntax
```

Additional issues and tickets related to callable symbolic expressions:

• #7512 fast_callable should respect the variable order in callable symbolic expressions (treating them like lambda functions rather than like expressions)
• #11507 make f(x,y,z)=vector make a vector-valued function
• #12075 Create callable matrices in function notation
• #12302 Partial evaluation bug for callable vector functions
• #15219 numerical integral needs an operand for callable symbolic functions
• #32008 `CallableSymbolicExpressionRing` over subrings of `SR`
• #32146 `CallableSymbolicExpressionRing`: Fix broken element methods
• #30374 Improve repr for callable symbolic expressions using unicode
• #31282 Allow to take the sign of a callable symbolic expression
• #30953 Implicit line continuation in callable symbolic expression

• #32227 Deprecate methods `arguments` (alias: `args`), `number_of_arguments`, `_fast_callable_` for non-callable symbolic expressions

comment:1 Changed 3 years ago by rburing

Description: modified (diff)

comment:2 follow-up:  3 Changed 3 years ago by jdemeyer

Let's look at these one by one:

1. The most obvious bug IMHO. Factoring in SR, especially in obvious cases like this, should work. This has nothing to do with callable expressions though.
2. Not really a bug, you shouldn't mix symbolic expressions and polynomials.
3. Bug. Substituting in matrices should work.
4. Not a bug, you cannot make a list symbolic.
5. Not a bug, the RHS `x.derivative()` equals 1 so `f(x) = x.derivative()` is just a complicated way to write `f(x) = 1`.
6. Bug, there is no reason why this shouldn't work.
7. I don't see the problem here, what would you expect?
8. Not a bug, similar to 5: the RHS `divisors(n)` is meaningless for a symbolic `n`.

Some of these (in particular 4, 5, 8) are really user mistakes where an ordinary Python function is meant instead of a symbolic function. I agree that this may be confusing, but there really is an important difference between

```f(x) = x.derivative()
```

and

```def f(x):
return x.derivative()
```

I recommend you to create separate tickets for the bugs that you want to see fixed. I cannot at all promise that they will be fixed. But it will help discussion if we don't have a single issue which is about multiple unrelated things.

comment:3 in reply to:  2 ; follow-up:  4 Changed 3 years ago by rburing

I have to disagree that these issues are unrelated. The main source of confusion (in my opinion) is that this syntax is reserved for something very particular (callable symbolic expressions), while it looks like something much more general, like a lambda-style definition of a Python function.

1. The most obvious bug IMHO. Factoring in SR, especially in obvious cases like this, should work. This has nothing to do with callable expressions though.

What I meant to show was: the (naive) expectation is that when you put in an integer, you get back an integer. This is false, because what you get is a symbolic expression. So a better example (with a method that isn't available on symbolic expressions) would be `f(2).binary()`.

1. Not really a bug, you shouldn't mix symbolic expressions and polynomials.

They can often be mixed. A "fix" here is to write `R(f(z+w)).coefficient({z : 1})`. My point is: how does the user know that `f(x) = x^2` defines a callable symbolic expression which should get this special treatment (compared to a Python function)? Or, how should a user know not to use these in this context?

1. Not a bug, you cannot make a list symbolic.

How does the user know that this is an issue? It's not an issue with Python functions...

1. Not a bug, the RHS `x.derivative()` equals 1 so `f(x) = x.derivative()` is just a complicated way to write `f(x) = 1`.

I agree that this may be confusing, but there really is an important difference between

```f(x) = x.derivative()
```

and

```def f(x):
return x.derivative()
```

We know this and we are used to it, but in my opinion it really makes no sense. I can kind of understand the desire to create a type of function object (which is what happens in the first case), e.g. for prettyprinting, but the fact that these two `f`'s don't give the same output when evaluated is honestly ridiculous. How should a user know?

1. I don't see the problem here, what would you expect?

Some would expect two scalar-valued functions of a scalar variable to always add up to a scalar-valued function of a scalar variable. But I will concede this is the least confusing one.

1. Not a bug, similar to 5: the RHS `divisors(n)` is meaningless for a symbolic `n`.

How does the user know that they are defining a symbolic callable expression, and that they shouldn't?

All the questions were rhetorical. In my experience (and in the posted threads), the user doesn't know, because it isn't clear. Sage would be less confusing without this syntax (in its current form).

comment:4 in reply to:  3 Changed 3 years ago by defeo

All the questions were rhetorical. In my experience (and in the posted threads), the user doesn't know, because it isn't clear. Sage would be less confusing without this syntax (in its current form).

So, is this ticket suggesting to remove the syntax? If so, I'm not convinced a ticket is the right place to have such a discussion.

Do you realize how much user code it would break? Maybe you should lay out your plan more clearly on sage-devel.

comment:5 follow-ups:  6  8 Changed 3 years ago by nbruin

Thanks for putting the list together. It's nice to have a record which issues (often?) arise. A lot of confusion happens because of the mathematics underneath. Confusion is a common state for students who are learning mathematics and I think it is to be expected that they will also encounter it while working with a computer algebra system of significant scope and complexity. It's nice to reduce confusion where possible, but I think it's unrealistic to expect we can do away with it.

Some of your items are indicated above as bugs and can probably be solved. Other items might be better to put in an FAQ document (or the documentation) for people to point to if questions arise. I don't know if these examples are best kept track of on a ticket. The ticket itself isn't really resolvable, but there are other administrative tracker tickets around.

Answer to "how does a user know"?

```sage: f(x)=x^2
sage: type(f)
<type 'sage.symbolic.expression.Expression'>
sage: parent(f)
Callable function ring with argument x
```

Once a user run into issues like this, it's clear that they have to progress beyond the level of "beginner" and start a more "advanced" approach. When they have questions concerning this, they have a motivation to read an "advanced" tutorial that goes more into the nuts and bolts of things. After that tutorial they should be at a level where they can make sense of the reference guide. Perhaps the "advanced" tutorial already exists. Perhaps it needs to be written?

Note that callable symbolic expressions serve a very definite purpose and are therefore not going away. The preparse trick of defining them `f(x)=...` is magic but incredibly useful for giving compact illustrations in educational settings, and that syntax doesn't seem to be a source for confusion anyway (other than that people don't get much of a cue that this is not a python built-in feature -- but letting people look at `preparse("...")` quickly illustrates what's going on. Callable symbolic expressions are necessary to use positional call syntax for symbolically defined functions/expressions, which is very common in math text books.

A general comment: Building a computer algebra system on top of python is necessarily a compromise. This is true for other python-based solutions too, for instance pandas or matplotlib. In most of these cases, a Domain Specific Language would be much more elegant and unified. Although, beware what you ask for: in for instance Maple, they DO have a very clear symbolic expression model that is fundamental to everything. And they found it doesn't address all their needs and had to bolt on other solutions. It makes for a decidedly unpleasant system to program for. They ended up with their compromises in other places.

1) symbolic factor COULD try and use integer factorization on objects that look like an integer or a rational number. That would change the complexity of the routine significantly (but perhaps for SR we don't care), and of course it might make people wonder why 2*I doesn't get factored.

5) doesn't seem to have to do with callable expressions at all. The fact that "derivative" is allowed to try to do something without an argument is necessary because of univariate calculus

6) Bug maybe, but possibly a consequence of design that will be hard to fix: "symbolic matrices" aren't members of the symbolic ring. They are matrices *over* SR. So filling them into a function that is from and to SR is problematic.

comment:6 in reply to:  5 ; follow-up:  7 Changed 3 years ago by kcrisman

Note that callable symbolic expressions serve a very definite purpose and are therefore not going away. The preparse trick of defining them `f(x)=...` is magic but incredibly useful for giving compact illustrations in educational settings, and that syntax doesn't seem to be a source for confusion anyway (other than that people don't get much of a cue that this is not a python built-in feature -- but letting people look at `preparse("...")` quickly illustrates what's going on.

Thanks for a very well-organized resume of the issues at stake, Nils.

comment:7 in reply to:  6 Changed 3 years ago by egourgoulhon

Thanks for a very well-organized resume of the issues at stake, Nils.

+1

comment:8 in reply to:  5 Changed 3 years ago by rburing

Thanks, Nils.

I had a look at the implementation. A callable symbolic expression like `f(x) = x^2` is mostly just an `Expression` whose parent is a `CallableSymbolicExpressionRing`; it keeps track of the list of (symbolic variable) arguments and implements `_call_element` (for substitution) and `_repr_element` (for prettyprinting). The notation `f(x,y) = (x,y)` defines a vector over a `CallableSymbolicExpressionRing`, and the special element class `Vector_callable_symbolic_dense` is used to fix the notation with a `_repr_` method.

This is quite simple and powerful, and I now agree that the notation should be kept. It seems that the best resolution is to fix the bugs, improve the documentation, improve error handling, and extend the functionality to include a matrix-valued variant. See a detailed proposal below.

I found that the Guided Tour already has a page Some Common Issues with Functions which is a great start for clarifying some of the issues. This should be the page to point people to when they run into any of this.

Let me go over the list again:

1) If `f(x) = x^2` then `f(2)` is not an `Integer`

The Common Issues page point 2 should explain more precisely what a callable symbolic expression is. In particular, at definition-time, the arguments are symbolic variables (and treated as such on the right-hand side); calling means substitution, and the result (in the scalar-valued case) is a symbolic expression. Also, the "vector-valued" (and possibly "matrix-valued", see 3 below) variant deserves a mention.

2) Polynomial confusion

Covered by the previous point. I would like to have the "solution" to this (explicit conversion) as an example somewhere, but since the Common Issues page is part of the Guided Tour it would be too early there.

3) Matrix confusion

Currently a bug. I believe matrices are not much different from vectors, and a matrix-valued variant could be implemented. It should return a matrix over a `CallableSymbolicExpressionRing`, and a special element class with `_repr_` and `_latex_` (like `Vector_callable_symbolic_dense`) should fix the notation.

4) List confusion

In my opinion the syntax with square brackets for vector-valued functions is an abuse of notation (which has caused confusion with lists); it should be `f(x) = (x,x)` or `f(x) = vector([x,x])` (the latter currently doesn't work, by the way). This is handled in `symbolic_expression`.

5) Derivative confusion (and argument confusion)

Handled by point 1. In particular, the point is that the arguments in the right-hand side of the definition of a callable symbolic expression are symbolic variables, not arbitrary objects, and in particular they cannot be functions.

6) Matrix argument confusion

From the implementation we know that `_call_element` is a substitution of symbolic variables. So all arguments must be (coerced to) symbolic expressions. Since this is not possible for a matrix, we can give a much better error here.

I can live with this. Maybe it should be mentioned in the documentation.

8) Non-symbolic function confusion

Can we improve the preparser here? Currently, it translates `tau(n) = len(divisors(n))` into

```__tmp__=var("n");
tau = symbolic_expression(len(divisors(n))).function(n)
```

```__tmp__=var("n");
try:
tau = symbolic_expression(len(divisors(n))).function(n)
except Exception, e:
raise TypeError, "Failed to define callable symbolic expression. Is the right-hand side a valid (tuple/vector/matrix of) symbolic expression(s) in symbolic variable(s) " + str(__tmp__) + "? Original exception: " + str(e)
```

The best part about this is that it informs the user that it is attempting to define a callable symbolic expression. If this is not what they meant, the user will probably do a web search for "Sage function" and will land on the (highly ranked) Common Issues page, which starts with the definition of ordinary Python functions.

comment:9 Changed 3 years ago by embray

Milestone: sage-8.9 → sage-9.1

Ticket retargeted after milestone closed

comment:10 Changed 3 years ago by mkoeppe

Milestone: sage-9.1 → sage-9.2

Batch modifying tickets that will likely not be ready for 9.1, based on a review of the ticket title, branch/review status, and last modification date.

comment:11 Changed 2 years ago by mkoeppe

Milestone: sage-9.2 → sage-9.3

comment:12 Changed 2 years ago by mkoeppe

Milestone: sage-9.3 → sage-9.4

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

comment:13 Changed 23 months ago by slelievre

Suggestion: make this a meta-ticket and track each item in its own ticket.

1. Derivative confusion.
```sage: f(t) = (t, t^2, t^3)

sage: f
t |--> (t, t^2, t^3)
sage: f(t)
(t, t^2, t^3)

sage: f.parent()
Vector space of dimension 3 over Callable function ring with argument t
sage: f(t).parent()
Vector space of dimension 3 over Symbolic Ring

sage: f1 = f(t).derivative(t)
sage: f1
(1, 2*t, 3*t^2)
sage: f1.parent()
Vector space of dimension 3 over Symbolic Ring

sage: g(t) = f(t).derivative(t)
Traceback (most recent call last)
...
TypeError: unable to convert (1, 2*t, 3*t^2) to a symbolic expression

sage: h = f.derivative()
sage: h
[    t |--> 1]
[  t |--> 2*t]
[t |--> 3*t^2]
sage: h.parent()
Full MatrixSpace of 3 by 1 dense matrices
over Callable function ring with argument t
```
1. Newline confusion. See #11621, #19088, #30953.
```sage: f(t) = (t,
....:         t^2,
....:         t^3)
File "<ipython-input-46-d381d3086d21>", line 2
t**Integer(2),
^
SyntaxError: invalid syntax
```

comment:14 Changed 19 months ago by mkoeppe

For item 7, see #32008.

comment:15 Changed 19 months ago by mkoeppe

Milestone: sage-9.4 → sage-9.5

comment:16 Changed 19 months ago by mkoeppe

Cc: mjo added modified (diff) Syntax for callable symbolic expressions causes too much confusion → Meta-ticket: Callable symbolic expressions

comment:17 Changed 19 months ago by mkoeppe

Description: modified (diff)

comment:18 Changed 19 months ago by mkoeppe

Description: modified (diff)

comment:19 Changed 19 months ago by mkoeppe

Description: modified (diff)

comment:20 Changed 14 months ago by mkoeppe

Milestone: sage-9.5 → sage-9.6

comment:22 Changed 11 months ago by mkoeppe

Description: modified (diff)

comment:23 Changed 10 months ago by mkoeppe

Milestone: sage-9.6 → sage-9.7

comment:24 Changed 5 months ago by mkoeppe

Milestone: sage-9.7 → sage-9.8

comment:25 Changed 4 weeks ago by mkoeppe

Milestone: sage-9.8 → sage-9.9
Note: See TracTickets for help on using tickets.