#11379 closed enhancement (fixed)
Add Quantumino solver to sage/games
Reported by:  slabbe  Owned by:  slabbe 

Priority:  major  Milestone:  sage4.7.2 
Component:  misc  Keywords:  sd31 
Cc:  Merged in:  sage4.7.2.alpha2  
Authors:  Sébastien Labbé  Reviewers:  Rob Beezer 
Report Upstream:  N/A  Work issues:  
Branch:  Commit:  
Dependencies:  Stopgaps: 
Description (last modified by )
Some code to solve the Quantumino Puzzle (see also this video on youtube).
Apply:
Attachments (10)
Change History (50)
comment:1 Changed 7 years ago by
comment:2 Changed 7 years ago by
Just added the patch.
The file takes 80 seconds to test... so I still need to add some "long" doctest warnings...
Sébastien
Changed 7 years ago by
comment:3 Changed 7 years ago by
 Status changed from new to needs_review
comment:4 Changed 7 years ago by
Takes now 17 seconds to test on my machine (35s with long test).
Changed 7 years ago by
comment:5 followup: ↓ 7 Changed 7 years ago by
 Status changed from needs_review to needs_work
Hi Sebastien,
Very nice  this is a lot of fun, and a great advertisement for the power of dancing links. Lots of little comments, I hope its not too much. Nothing very serious.
 One doctest failure. I'm on 64bit Linux  could 32bit/64bit be the cause?
sage t "devel/sagemain/sage/games/quantumino.py" ********************************************************************** File "/sage/sage4.7/devel/sagemain/sage/games/quantumino.py", line 193: sage: hash(p) Expected: 2059134902 Got: 6915256369230374838 **********************************************************************
 Why does block number 8 (yellow) have a hole in the middle? See discussion below about
size
.
 Documentation looks good. I would have liked to see a bit more detail at the module level for a quickstart  more like for the
QuantuminoSolver
classlevel documentation. Specifically: Make it clearer that you will see the puzzle pieces in 3D with the
show_pentaminos()
command.
 Show how to get the list of the actual placements for a single solution. I wanted to do this once I had a solution since the
_repr_
was not as informative as my curiosity.  Maybe a real quick overview of the classes: the pentamino and polyomino classes, the solution class, and the solver class.
 Make it clearer that you will see the puzzle pieces in 3D with the
 I thought an interact would be fun. Checkboxes for the excluded piece, plus the ability to "explode" the solution via a slider. I'm not suggesting you write an interact, but a
size
argument as input toshow3d()
(for the solution) would make this possible. Patch attempts to do this, but it is not totally correct, at size below 0.50 the pieces start to fall apart into cubes. And the aside piece breaks up even earlier in my test. The hole in block number 8 behaves slightly differently. So as a suggestion: consider adding asize
argument to pass from the solutionshow3d()
down to each piece. But my patch is just a suggestion  it is not ready to use.
 Some of the lesser methods could use some improved documentation. In many cases, at least the first summary line, and/or the type of input. For example:
 "Return the 3D polyomino defined by a set of 3d coordinates." It is not clear what "defined by" means here, maybe the coordinates are for the "bottommost" corner of each constituent cube.
 It is not clear what sub does without looking at the code. A oneline summary and an input list would be enough, I think.
 I'm not sure uppercase is part of Python or Sage style for code. SPACE, COORD_TO_INT, and INT_TO_COORD look funny to my eye.
 I set up Sudoku puzzles with a "Sudoku" class. Then s = Sudoku(.....), followed by s.solve() gave an iterator over solutions. Would it be worth trying to mimic that approach for consistency? I'm not arguing that my approach was better  just first.
:)
Minor editing
 "needs to creates" should be "create"
 "where each pieces is used exactly once" should be "piece"
# Class QuantuminoSolultion
(misspelled in comment) I think even with the utf8 header the consensus (rule?) has been straight ASCII in source files? Which I know even impacts your name. I wish it wasn't this way (maybe I'm wrong?). The trademark symmbol and bracketing on "think inside the box" for the game description would need adjustments.

#bug trac #11272
 can this go away? 
#return G
(twice)  can these go away?
As I said, lots of little stuff, which I hope does not look like too long a list. I've tried to keep it to suggestions so you have the latitude to approach it as you see fit. I'll be happy to stick with this review as you make revisions.
Rob
comment:6 Changed 7 years ago by
One more comment I forgot:
ss=quantumino_solver.get_solution(7, box=(3,3,9))
seems to totally hang on my system  not even a CtrlC gets it back. Do I need to use a box of exactly volume 80? Was it unreasonable to expect a different result?
comment:7 in reply to: ↑ 5 ; followup: ↓ 9 Changed 7 years ago by
Hi Rob,
Thanks a lot for your good review. I almost done making the corrections. I also added a class Tiling Solver which replaces the puzzle solver function. This allows for more introspection (for instance looking at the rows passed to the DLX solver) and also comptute the number of solutions more efficiently. I have one question about your suggestion :
 I thought an interact would be fun. Checkboxes for the excluded piece, plus the ability to "explode" the solution via a slider. I'm not suggesting you write an interact, but a
size
argument as input toshow3d()
(for the solution) would make this possible. Patch attempts to do this, but it is not totally correct, at size below 0.50 the pieces start to fall apart into cubes. And the aside piece breaks up even earlier in my test. The hole in block number 8 behaves slightly differently. So as a suggestion: consider adding asize
argument to pass from the solutionshow3d()
down to each piece. But my patch is just a suggestion  it is not ready to use.
I don't understand what is meant by ""explode" the solution". Is this a slider which would bring the size of the cube from 0 to 1 ? It doesn't not seem that nice to me. I would rather suggest a slider which would go from one solution to the other, where we would see the pieces that are removed and added, etc. Or maybe even an animation of it. I am almost done doing it: I have the iterator of partial solutions. I only need to know how to make a Jmol animation of 3D Graphics object or maybe an animation of Tachyon images.
Sébastien
comment:8 Changed 7 years ago by
 Description modified (diff)
comment:9 in reply to: ↑ 7 Changed 7 years ago by
 Status changed from needs_work to needs_review
I don't understand what is meant by ""explode" the solution".
Ok, now I understand. I needed to try it ! It is true that it helps to change the size of the small cubes. Because of conflicts, I had to reload your patch (but could not erase yours) so I renamed it. Hence, your patch apply over my second patch.
I think I was able to answer all of your comments. Needs review!
Sébastien
comment:10 followup: ↓ 11 Changed 7 years ago by
I had a bit of time to look through the patch. Looks great! I still need to do a thorough test of the new features and all, so will try to get to that soon.
Would Franco let you bring the real physical puzzle to Seattle for SD 31?
Rob
comment:11 in reply to: ↑ 10 Changed 7 years ago by
Would Franco let you bring the real physical puzzle to Seattle for SD 31?
Sure! I'll ask. And will try not to forget it!
Sébastien
comment:12 Changed 7 years ago by
 Component changed from PLEASE CHANGE to misc
comment:13 followup: ↓ 14 Changed 7 years ago by
 Status changed from needs_review to needs_work
I have been working on it again yesterday. I will update the patches again quite soon. Do not review until then.
Sebastien
comment:14 in reply to: ↑ 13 Changed 7 years ago by
 Description modified (diff)
 Keywords sd31 added
 Reviewers set to Rob Beezer
Replying to slabbe:
I have been working on it again yesterday. I will update the patches again quite soon. Do not review until then.
Thanks  ready whenever you are (I think!).
comment:15 Changed 7 years ago by
Great. So I will upload the patches later tonight! I still have some cleaning to make.
comment:16 Changed 7 years ago by
 Description modified (diff)
 Status changed from needs_work to needs_review
 Summary changed from Add Quantamino solver to sage/games to Add Quantumino solver to sage/games
Ok, so I just reuploaded the correction patch. The size suggestion patch as been folded into that correction patch. So only two patches are needed to be applied (the one that has already been reviewed and the correction patch).
So, compared to what has already been reviewed, I did a bunch of improvements: I created a new file sage/combinat/tiling.py
and moved the polyomino class into it. Also, I created a new class called TilingSolver
which solves the general problem of Tiling a box by polyomino. This class replaces the old function general_puzzle_solver
which I might misspell. The TilingSolver
class allows to do more introspection like getting the rows passed to the DLX solver and count them. One can also get the DLX Solver. I managed to write the Polyomino
and TilingSolver
abstract enough so that they can be defined in any dimension. Ploting works when the dimension is 2 or 3. I also added parameters to allow (or not) reflections and rotations and whether the pieces can be reused or not.
There is still one issue mentionned in the review that I did not fixed. The holes in the polyomino. Maybe tomorrow we can think about a efficient way to fix this?
Question: Should I use Pentomino like Donald Knuth does or Pentamino like the game Quantumino calls the pieces? Which is best?
Good night!
comment:17 Changed 7 years ago by
 Description modified (diff)
comment:18 Changed 7 years ago by
 Description modified (diff)
In a new patch I just uploaded which applies on top of two others, I fixed a color issue for when pieces are reusable. Sorry, I had the idea for the fix when I woke up. Now, I stop working on it!
comment:19 Changed 7 years ago by
Ok, so I can't stop working on it apparently. I added the possibility of making an animation. I also fixed the problem that showed up during my quick demo.
Needs review!
Changed 7 years ago by
comment:20 followups: ↓ 21 ↓ 23 Changed 7 years ago by
 Description modified (diff)
Sebastien,
Sorry to be so tardy on this one. Really just one "issue" that I think needs attention.
Made a reviewer patch: Changed some module and class links in documentation to be active, fixed a couple minor English language things. Do not list me as an author for these.
The animations are great. Can you do something to mark the end (like a few blank frames, for maybe a halfsecond)? It goes so fast, it is hard to tell where the start is and where the end is.
Related Questions:
 Pentamino 8 (a yellow one) still has a hole in it. Not a big deal, but perhaps a symptom of something that should be done more carefully?
 I feel bad to bring this up, since I suggested it. The "size" parameter works fine for 0.5 < size < 1.0. Proably size > 1.0 should raise an error. Below size=0.5 the puzzle pieces seem to break into lots of individual cubes. Also the hole in piece 8 seems to change according to size. So I wonder if these first two items are related.
 In the 3D puzzle in the tiling module documentation, there are again holes in the pieces.
 Maybe each cube of a piece needs to be shrunk and translated, relative to some anchor point (the "corner" closest to the origin?). I have not looked carefully enough to know if this what needs to be done.
Rob
comment:21 in reply to: ↑ 20 ; followup: ↓ 22 Changed 7 years ago by
 Status changed from needs_review to needs_work
Hi Rob,
Sorry to be so tardy on this one. Really just one "issue" that I think needs attention.
No problem!
Made a reviewer patch: Changed some module and class links in documentation to be active, fixed a couple minor English language things. Do not list me as an author for these.
Good, thanks for those fixes! Is there a problem with one of the fixes ? Because there is a symbol tilde ~
that appears in front of one of the class path.
:class:`~sage.combinat.tiling.TilingSolver`
The animations are great. Can you do something to mark the end (like a few blank frames, for maybe a halfsecond)? It goes so fast, it is hard to tell where the start is and where the end is.
The delay between frames and the number of iterations are arguments of the method show (see animate??
). I would keep the animation function as is, but add an exemple in the doctest of how to change those parameters and add blank frames at the end. What do you think?
 Pentamino 8 (a yellow one) still has a hole in it. Not a big deal, but perhaps a symptom of something that should be done more carefully?
Ok. I will think about it. Let me find a solution which will be better than the "cube in the middle" I am using up to now. Maybe using Simplicial Complexes of cubes, I could get the exact boundary of the piece? I take a look at it and comes back with a fix soon.
Sébastien
comment:22 in reply to: ↑ 21 Changed 7 years ago by
Replying to slabbe:
No problem!
Good. We just need to finish this before you become a father. ;)
:class:`~sage.combinat.tiling.TilingSolver`
The tilde should suppress the sage.combinat.tiling
prefix in the output. Don't remember just why I did it that way there (I'm sure I had a reason at the time!)  but you should feel free to adjust it if you would rather have the fullyqualified name.
The delay between frames and the number of iterations are arguments of the method show (see
animate??
). I would keep the animation function as is, but add an exemple in the doctest of how to change those parameters and add blank frames at the end. What do you think?
Perfect.
Ok. I will think about it. Let me find a solution which will be better than the "cube in the middle" I am using up to now. Maybe using Simplicial Complexes of cubes, I could get the exact boundary of the piece? I take a look at it and comes back with a fix soon.
Maybe if each piece had a center, or maybe a center of a bounding box, or something like that. Then you could shrink into the center. Seems like you work off a corner right now, but again, I have not studied it very carefully. If the size parameter becomes too much trouble, feel free to drop it, but I think fixing this will fix a variety of other things, like the hole in piece 8.
comment:23 in reply to: ↑ 20 ; followup: ↓ 28 Changed 7 years ago by
 Status changed from needs_work to needs_review
I just uploaded a new patch which applies over the precedent ones.
The animations are great. Can you do something to mark the end (like a few blank frames, for maybe a halfsecond)? It goes so fast, it is hard to tell where the start is and where the end is.
I added a paragraph saying (copied from the animate doc string) :
The ``show`` function takes arguments to specify the delay between frames (measured in hundredths of a second, default value 20) and the number of iterations (default value 0, which means to iterate forever). To iterate 4 times with half a second between each frame:: sage: a.show(delay=50, iterations=4) # optional
I also fixed the methods dlx_common_prefix_solutions
and dlx_incremental_solutions
which were broken. That was maybe the reason why the animations were going so fast... So now, the following animation looks better and not too fast even with default parameters of the method show :
sage: from sage.combinat.tiling import Polyomino, TilingSolver sage: y = Polyomino([(0,0),(1,0),(2,0),(3,0),(2,1)], color='cyan') sage: T = TilingSolver([y], box=(5,10), reusable=True, reflection=True) sage: a = T.animate('incremental') sage: a Animation with 123 frames sage: a.show()
 Maybe each cube of a piece needs to be shrunk and translated, relative to some anchor point (the "corner" closest to the origin?).
Ok. So I implemented this solution (translation to origin, shrinked, translated back) and changed show2d and show3d methods accordingly. For the Quantumino, it looks great. The yellow pentamino number 8 do not have a hole anymore :
sage: from sage.games.quantumino import show_pentaminos sage: show_pentaminos()
Also, using size<0.5 does not create disconnected cubes :
sage: from sage.games.quantumino import QuantuminoSolver sage: s = QuantuminoSolver(0).solve().next() sage: s.show3d(size=0.3)
Although, I can not say that the proposed solution is perfect and always better than the precedent one. In the example below, there are no holes anymore which is good. But, the space between each piece is not uniform and it is even hard to find a size which will avoid the pieces to touch each other without being to far from each other:
sage: from sage.combinat.tiling import Polyomino, TilingSolver sage: L = [] sage: L.append(Polyomino([(0,0),(0,1),(0,2),(0,3),(1,0),(1,1),(1,2),(1,3)])) sage: L.append(Polyomino([(0,0),(0,1),(0,2),(0,3),(1,0),(1,1),(1,2)])) sage: L.append(Polyomino([(0,0),(0,1),(0,2),(0,3),(1,0),(1,1),(1,3)])) sage: L.append(Polyomino([(0,0),(0,1),(0,2),(0,3),(1,0),(1,3)])) sage: L.append(Polyomino([(0,0),(0,1),(0,2),(0,3),(1,0),(1,1)])) sage: L.append(Polyomino([(0,0),(0,1),(0,2),(0,3),(1,1),(1,2)])) sage: L.append(Polyomino([(0,0),(0,1),(0,2),(0,3),(1,1),(1,3)])) sage: L.append(Polyomino([(0,1),(0,2),(0,3),(1,0),(1,1),(1,3)])) sage: L.append(Polyomino([(0,1),(0,2),(0,3),(1,0),(1,1),(1,2)])) sage: L.append(Polyomino([(0,0),(0,1),(0,2),(1,0),(1,1),(1,2)])) sage: T = TilingSolver(L, (8,8), reflection=True) sage: solution = T.solve().next() sage: G = sum([piece.show2d(size=0.85) for piece in solution], Graphics()) sage: G.show(aspect_ratio=1)
What do you think?
Sébastien
comment:24 Changed 7 years ago by
 Description modified (diff)
comment:25 Changed 7 years ago by
 Description modified (diff)
I don't know how to speak to the buildbot. Maybe the commas between patches name are needed? I am trying.
More info is here : http://wiki.sagemath.org/buildbot
comment:26 Changed 7 years ago by
Or maybe the information must be put in a comment instead of above in the description :
For the patchbot:
Apply trac_11379_quantaminosl.patch, trac_11379_correctionssl.patch, trac_11379_color_issuesl.patch, trac_11379reviewerdocs.patch, trac_11379_hole_bugsl.patch
comment:27 Changed 7 years ago by
 Description modified (diff)
For the patchbot:
Apply trac_11379_quantaminosl.patch, trac_11379_correctionssl.patch, trac_11379_color_issuesl.patch, trac_11379reviewerdocs.patch, trac_11379_hole_bugsl.patch, trac_11379_2d_boundarysl.patch
comment:28 in reply to: ↑ 23 Changed 7 years ago by
Replying to slabbe:
Although, I can not say that the proposed solution is perfect and always better than the precedent one.
I change the way to show 2d polyomino. First, I reverted its drawing as it was before. Second, I added a boundary line. Thirdly, I made the edge between adjacent points smaller than before. This way, holes are more esthetic and natural : we accept them more easily.
You can see the result with this example :
sage: from sage.combinat.tiling import Polyomino, TilingSolver sage: L = [] sage: L.append(Polyomino([(0,0),(0,1),(0,2),(0,3),(1,0),(1,1),(1,2),(1,3)], 'yellow')) sage: L.append(Polyomino([(0,0),(0,1),(0,2),(0,3),(1,0),(1,1),(1,2)], "black")) sage: L.append(Polyomino([(0,0),(0,1),(0,2),(0,3),(1,0),(1,1),(1,3)], "gray")) sage: L.append(Polyomino([(0,0),(0,1),(0,2),(0,3),(1,0),(1,3)],"cyan")) sage: L.append(Polyomino([(0,0),(0,1),(0,2),(0,3),(1,0),(1,1)],"red")) sage: L.append(Polyomino([(0,0),(0,1),(0,2),(0,3),(1,1),(1,2)],"blue")) sage: L.append(Polyomino([(0,0),(0,1),(0,2),(0,3),(1,1),(1,3)],"green")) sage: L.append(Polyomino([(0,1),(0,2),(0,3),(1,0),(1,1),(1,3)],"magenta")) sage: L.append(Polyomino([(0,1),(0,2),(0,3),(1,0),(1,1),(1,2)],"orange")) sage: L.append(Polyomino([(0,0),(0,1),(0,2),(1,0),(1,1),(1,2)],"pink")) sage: T = TilingSolver(L, (8,8), reflection=True) sage: solution = T.solve().next() sage: G = sum([piece.show2d() for piece in solution], Graphics()) sage: G.show(aspect_ratio=1, axes=False)
Or this animation :
sage: a = T.animate() #45 seconds sage: a Animation with 328 frames sage: a.show() # take some time like 2 minutes
Now, I am happy with the patch. Needs review!
Sébastien
comment:29 Changed 7 years ago by
The above 328 frames animation is here :
http://thales.math.uqam.ca/~labbes/Experimentations/florent.gif
done with the following parameters
sage: a.show(delay=50, iterations=1)
comment:30 Changed 7 years ago by
 Status changed from needs_review to positive_review
I like all the changes (no holes!). And the 2D pieces look real nice. I think this is ready to go  builds, passes long tests, nice documentation on 4.7.1.alpha4. So positive review.
One patch needed a commit message, and since it was easy, I just rolled everything into one big "total" patch. Still has Sebastian's name on it.
Nice work on a big project  this will be a great way to demonstrate backtracking (and dancing links).
Rob
comment:31 Changed 7 years ago by
 Milestone changed from sage4.7.1 to sage4.7.2
comment:32 Changed 7 years ago by
 Status changed from positive_review to needs_work
The commit message of trac_11379_correctionssl.patch should be changed since it contains a reference to a mercurial queue.
comment:33 followup: ↓ 35 Changed 7 years ago by
 Description modified (diff)
 Status changed from needs_work to positive_review
I consolidated the commit strings ont eh one patch, and had to use a new name for the file, since I do not have the privileges to replace it.
Rob
comment:34 Changed 7 years ago by
 Merged in set to sage4.7.2.alpha2
 Resolution set to fixed
 Status changed from positive_review to closed
comment:35 in reply to: ↑ 33 ; followup: ↓ 36 Changed 7 years ago by
Replying to rbeezer:
I consolidated the commit strings ont eh one patch, and had to use a new name for the file, since I do not have the privileges to replace it.
Thanks Rob for taking care of this. Thanks again for the whole review. I am happy to see this is now merged!
Sébastien
comment:36 in reply to: ↑ 35 Changed 7 years ago by
Replying to slabbe:
Replying to rbeezer: Thanks Rob for taking care of this.
No problem. I knew Jeroen was busy merging patches, and I was busy rebasing some of my linear algebra stuff, so I wanted to "strike while the iron was hot."
Thanks again for the whole review. I am happy to see this is now merged!
My pleasure  it is a nice piece of work. Feel free to cc me on other combinatorial goodness.
Best of luck with the upcoming addition to the family!
Rob
comment:37 followup: ↓ 38 Changed 7 years ago by
Just a note from my testing of 4.7.2.alpha2. I have no problem on a vanilla sage even with python 2.7 installed. However in sageongentoo one test is failing
.. NOTE:: The DLX solver throws a Segmentation Fault when the number of rows is zero:: sage: from sage.combinat.matrices.dancing_links import dlx_solver sage: rows = [] sage: x = dlx_solver(rows) sage: x.search() # not tested BOOM !!!
sageongentoo goes BOOM at
sage: x = dlx_solver(rows)
With a SIGABRT
sage: x = dlx_solver(rows) python2.7: sage/combinat/matrices/dancing_links_c.h:217: void dancing_links::setup_columns(): Assertion `nr_columns > 0' failed. /usr/lib64/libcsage.so(print_backtrace+0x24)[0x7fea73a945f7] /usr/lib64/libcsage.so(sigdie+0x1d)[0x7fea73a94687] /usr/lib64/libcsage.so(sage_signal_handler+0x157)[0x7fea73a94822] /lib64/libpthread.so.0(+0xfee0)[0x7fea79206ee0] /lib64/libc.so.6(gsignal+0x35)[0x7fea78ea5ee5] /lib64/libc.so.6(abort+0x186)[0x7fea78ea7896] /lib64/libc.so.6(__assert_fail+0xf5)[0x7fea78e9e7a5] /usr/lib64/python2.7/sitepackages/sage/combinat/matrices/dancing_links.so(+0x855d)[0x7fea4e73255d]
while normal sage goes BOOM in the next not tested bit with a SIGSEGV
sage: x = dlx_solver(rows) sage: x.search() /home/work/fbissey/sandbox/sage4.7.2.alpha2/local/lib/libcsage.so(print_backtrace+0x31)[0x7fe2bbd11e65] /home/work/fbissey/sandbox/sage4.7.2.alpha2/local/lib/libcsage.so(sigdie+0x14)[0x7fe2bbd11e97] /home/work/fbissey/sandbox/sage4.7.2.alpha2/local/lib/libcsage.so(sage_signal_handler+0x20e)[0x7fe2bbd11ae4] /lib64/libpthread.so.0(+0xfee0)[0x7fe2c0fc5ee0] /home/work/fbissey/sandbox/sage4.7.2.alpha2/local/lib/python2.7/sitepackages/sage/combinat/matrices/dancing_links.so(+0x847d)[0x7fe29b0c547d]
We go BOOM in sageongentoo for the reason stated in the notes (0 rows).
comment:38 in reply to: ↑ 37 Changed 7 years ago by
sageongentoo goes BOOM at
sage: x = dlx_solver(rows)
Would adding # not tested
at the end of that line would be an acceptable fix ?
Should we open a ticket for it?
Sébastien
comment:39 Changed 7 years ago by
I am doing that right now in sageongentoo. I suspect that in a vanilla sage install
sage: x = dlx_solver(rows)
is left unevaluated and you only get a segfault when accessing x. The only thing that tells me that something else may happen is that I get a SIGABRT instead of a SIGSEGV. I don't think we need to open a new ticket or change this one for that matter. sageongentoo doesn't have official status. I wanted to make a comment in case it prompted in insight. I suspect it may surface in vanilla sage sooner or later but it is difficult to know what is the trigger since I am applying a few tickets on top of 4.7.2.alpha2 plus a few custom hacks here and there.
comment:40 Changed 7 years ago by
Any Segmentation Fault, documented or not, is a bug. So I think this really points to a problem with the original patch. See #11814 for a followup.
I saw this demo'ed at Sage Days 30. It's really nice. I'll give the patch as look once it appears.