Opened 10 years ago
Closed 6 months ago
#11736 closed enhancement (fixed)
Linear time implementation of lex_BFS()
Reported by:  ddestrada  Owned by:  jason, ncohen, rlm 

Priority:  major  Milestone:  sage9.2 
Component:  graph theory  Keywords:  lexbfs 
Cc:  dcoudert, ghkliem, mjo  Merged in:  
Authors:  Diego de Estrada, David Coudert  Reviewers:  Michael Orlitzky 
Report Upstream:  N/A  Work issues:  
Branch:  332fb8b (Commits, GitHub, GitLab)  Commit:  332fb8bf3acb23d8cb1a376744665cd25c680c26 
Dependencies:  Stopgaps: 
Description (last modified by )
The current implementation of lex_BFS() is quadratic, which is not optimal. The usual way to do it in linear time is through a clever use of doubly linked lists, but in (1) there is a way with static arrays, which I coded in Python.
There is one thing, though: in (2) there is a new algorithm for is_interval() that avoids PQtrees, and just uses various passes of lex_BFS() (implemented with doubly linked lists). So maybe in the long run it would be better to do it with doubly linked lists in Cython.
(1) Habib, McConnell, Paul and Viennot. LexBFS and Partition Refinement, with Applications to Transitive Orientation, Interval Graph Recognition and Consecutive Ones Testing. TCS 234, 2000.
(2) Corneil, Olariu and Stewart. The LBFS Structure and Recognition of Interval Graphs. SIAMDM 23(4), 2009.
Attachments (2)
Change History (51)
comment:1 Changed 10 years ago by
 Owner changed from jason, ncohen, rlm to ddestrada
 Type changed from PLEASE CHANGE to enhancement
comment:2 Changed 10 years ago by
 Owner changed from ddestrada to jason, ncohen, rlm
comment:3 Changed 10 years ago by
 Description modified (diff)
comment:4 followup: ↓ 6 Changed 10 years ago by
 Status changed from new to needs_review
comment:5 Changed 10 years ago by
 Status changed from needs_review to needs_work
Changed 9 years ago by
comment:6 in reply to: ↑ 4 ; followup: ↓ 7 Changed 9 years ago by
 Status changed from needs_work to needs_review
Replying to ncohen: Thank you! Here I fixed the "build/" thing, but I'm having trouble running the tests (in OSX 10.7 and Ubuntu 11.04), so expect it to break some of them. I'm sorry.
comment:7 in reply to: ↑ 6 ; followup: ↓ 8 Changed 9 years ago by
 Status changed from needs_review to needs_work
Thank you! Here I fixed the "build/" thing, but I'm having trouble running the tests (in OSX 10.7 and Ubuntu 11.04), so expect it to break some of them. I'm sorry.
Perhaps I can help you with this test problem ? The thing is that right now the patch does not seem to work :
sage: graphs.RandomInterval(30).is_interval()  TypeError Traceback (most recent call last) /home/ncohen/sage/devel/sage3/sage/graphs/<ipython console> in <module>() /home/ncohen/sage/local/lib/python2.6/sitepackages/sage/graphs/generic_graph.pyc in is_interval(self, certificate) 9606 # by the way :) 9607 > 9608 if not self.is_chordal(): 9609 return False 9610 /home/ncohen/sage/local/lib/python2.6/sitepackages/sage/graphs/generic_graph.pyc in is_chordal(self, certificate) 9492 for v in peo: 9493 > 9494 if t_peo.out_degree(v)>0: 9495 9496 x = t_peo.neighbor_out_iterator(v).next() /home/ncohen/sage/local/lib/python2.6/sitepackages/sage/graphs/digraph.pyc in out_degree(self, vertices, labels) 1176 return di 1177 else: > 1178 return list(self.out_degree_iterator(vertices, labels=labels)) 1179 1180 def out_degree_iterator(self, vertices=None, labels=False): /home/ncohen/sage/local/lib/python2.6/sitepackages/sage/graphs/digraph.pyc in out_degree_iterator(self, vertices, labels) 1221 yield (v, d) 1222 else: > 1223 for v in vertices: 1224 d = 0 1225 for e in self.outgoing_edge_iterator(v): TypeError: 'int' object is not iterable sage: graphs.RandomInterval(30).is_interval()  IndexError Traceback (most recent call last) /home/ncohen/sage/devel/sage3/sage/graphs/<ipython console> in <module>() /home/ncohen/sage/local/lib/python2.6/sitepackages/sage/graphs/generic_graph.pyc in is_interval(self, certificate) 9606 # by the way :) 9607 > 9608 if not self.is_chordal(): 9609 return False 9610 /home/ncohen/sage/local/lib/python2.6/sitepackages/sage/graphs/generic_graph.pyc in is_chordal(self, certificate) 9484 return all( gg.is_chordal() for gg in self.connected_components_subgraphs() ) 9485 > 9486 peo,t_peo = self.lex_BFS(tree=True) 9487 9488 g = self.copy() /home/ncohen/sage/local/lib/python2.6/sitepackages/sage/graphs/generic_graph.pyc in lex_BFS(self, reverse, tree, initial_vertex) 11617 if subslice[a] < old_k: 11618 subslice[a] = k > 11619 slice_head[k] = l 11620 k += 1 11621 slice_of[l] = subslice[a] IndexError: list assignment index out of range
On random interval graphs sometimes the is_interval function works, sometimes it does not... ^^;
I tested you patch after having applied #11738 and #11735 but I do not think this is where the bug comes from. Could you also add some comments to your code, along with a reference to the paper you are following for this implementation ? :)
Nathann
comment:8 in reply to: ↑ 7 Changed 9 years ago by
 Status changed from needs_work to needs_review
Replying to ncohen: Hello Nathann, I think the bug is now fixed (it was a very simple mistake). I added the reference, but the comments are still missing. Thanks!
comment:9 Changed 9 years ago by
 Status changed from needs_review to needs_work
Hello !!
This patch still produces many errors when running the doctests...
sage t "devel/sage3/sage/graphs/generic_graph.py" IBM ILOG License Manager: "IBM ILOG Optimization Suite for Academic Initiative" is accessing CPLEX 12 with option(s): "e m b q ". ********************************************************************** File "/home/ncohen/.Sage/devel/sage3/sage/graphs/generic_graph.py", line 9412: sage: (2*g).is_chordal() Exception raised: Traceback (most recent call last): File "/home/ncohen/.Sage/local/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) File "/home/ncohen/.Sage/local/bin/sagedoctest.py", line 38, in run_one_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags) File "/home/ncohen/.Sage/local/bin/ncadoctest.py", line 1172, in run_one_example compileflags, 1) in test.globs File "<doctest __main__.example_137[6]>", line 1, in <module> (Integer(2)*g).is_chordal()###line 9412: sage: (2*g).is_chordal() File "/home/ncohen/.Sage/local/lib/python/sitepackages/sage/graphs/generic_graph.py", line 9474, in is_chordal return all( gg.is_chordal() for gg in self.connected_components_subgraphs() ) File "/home/ncohen/.Sage/local/lib/python/sitepackages/sage/graphs/generic_graph.py", line 9474, in <genexpr> return all( gg.is_chordal() for gg in self.connected_components_subgraphs() ) File "/home/ncohen/.Sage/local/lib/python/sitepackages/sage/graphs/generic_graph.py", line 9488, in is_chordal if t_peo.out_degree(v)>0 and g.neighbors(v) not in neighbors_subsets[t_peo.neighbor_out_iterator(v).next()]: File "/home/ncohen/.Sage/local/lib/python/sitepackages/sage/graphs/digraph.py", line 1178, in out_degree return list(self.out_degree_iterator(vertices, labels=labels)) File "/home/ncohen/.Sage/local/lib/python/sitepackages/sage/graphs/digraph.py", line 1223, in out_degree_iterator for v in vertices: TypeError: 'int' object is not iterable ********************************************************************** File "/home/ncohen/.Sage/devel/sage3/sage/graphs/generic_graph.py", line 9436: sage: hole Expected: Subgraph of (Petersen graph): Graph on 5 vertices Got: Subgraph of (Petersen graph): Graph on 4 vertices ********************************************************************** File "/home/ncohen/.Sage/devel/sage3/sage/graphs/generic_graph.py", line 9438: sage: hole.is_isomorphic(graphs.CycleGraph(5)) Expected: True Got: False ********************************************************************** File "/home/ncohen/.Sage/devel/sage3/sage/graphs/generic_graph.py", line 9554: sage: g.is_interval() Expected: True Got: False ********************************************************************** File "/home/ncohen/.Sage/devel/sage3/sage/graphs/generic_graph.py", line 9562: sage: d = g.is_interval(certificate = True) Exception raised: Traceback (most recent call last): File "/home/ncohen/.Sage/local/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) ...
Perhaps we can help with your problem running doctests ? It something does not work on this side, it is another bug of Sage's ! ;)
Nathann
comment:10 followup: ↓ 11 Changed 9 years ago by
I was ready to help with the doctests issues, but I'm unable to install the patch on sage5.0.beta9.
comment:11 in reply to: ↑ 10 Changed 9 years ago by
Replying to dcoudert:
I was ready to help with the doctests issues, but I'm unable to install the patch on sage5.0.beta9.
Hello! I'm sorry I disappeared. I will fix this in the next few days.
Changed 9 years ago by
comment:12 Changed 9 years ago by
 Status changed from needs_work to needs_review
comment:13 Changed 9 years ago by
 Cc dcoudert added
comment:14 followup: ↓ 15 Changed 9 years ago by
I think this patch is working now and passes all the doctests. I'm sure the errors I get while doctesting are bugs of Sage 4.8 (for example, on both Ubuntu 10.10 and OS X 10.7 I get an error in line 15146 of generic_graph.py: "ValueError?: Database has no tablegraph_data.").
comment:15 in reply to: ↑ 14 Changed 9 years ago by
 Status changed from needs_review to needs_work
Helloooooo Diego !!
I'm sorry but your patch can not be applied on top of the latest release of Sage, that is sage5.0beta11 (https://groups.google.com/d/topic/sagerelease/XnyN3tx4Bow/discussion). Plenty of stuff has been added in the 5.0beta? series, and that is probably the explanation :)
About your path, though : it removes the old code that computed (more slowly) a BFS ordering for the vertices. This being said, I think this code could still be useful to ... check that the new algorithm actually returns a correct BFS ordering ! Of course, as it is slower, this ordering should not be checked by default, but it would be nice to be able to run doctests which check for each computation of BFS ordering whether this ordering is indeed correct. This can be done by actually checking in turn that each vertex of the ordering is maximal at the moment when it is removed.
Do you think it could be possible to comment your code a bit more ? As it is, reading the paper to understand what happens seems to be the only alternative :)
Nathann
comment:16 Changed 8 years ago by
 Milestone changed from sage5.11 to sage5.12
comment:17 Changed 7 years ago by
 Milestone changed from sage6.1 to sage6.2
comment:18 Changed 7 years ago by
 Milestone changed from sage6.2 to sage6.3
comment:19 Changed 7 years ago by
 Milestone changed from sage6.3 to sage6.4
comment:20 Changed 6 years ago by
 Branch set to public/11736
comment:21 Changed 6 years ago by
 Commit set to 588c43140644c9d43cfae2e27e116566eda9ac54
Branch pushed to git repo; I updated commit sha1. New commits:
588c431  trac #11736: convert hg patch to git

comment:22 Changed 6 years ago by
 Status changed from needs_work to needs_info
Hello,
This patch has been sleeping for very very long time.
I did a first git version of the patch with some corrections. This is far from final version and we have currently 2 implementations to ease tests. To use the new version, you need to set parameter new_version=True
.
Since the code has no comments, it is hard to know whats going on. I have corrected at least one bug (a missing j = l
), but I have the following issue:
sage: g = graphs.PathGraph(3).lexicographic_product(graphs.CompleteGraph(2)) sage: g.lex_BFS(reverse=True) [(2, 1), (2, 0), (1, 1), (1, 0), (0, 1), (0, 0)] sage: g.lex_BFS(reverse=True, new_version=True) [(2, 1), (2, 0), (1, 1), (1, 0), (0, 1), (0, 0)] sage: g.lex_BFS(reverse=False) [(0, 0), (0, 1), (1, 0), (1, 1), (2, 0), (2, 1)] sage: g.lex_BFS(reverse=False, new_version=True) [(0, 0), (0, 1), (1, 0), (1, 1), (2, 0), (2, 1)]
The new version is more consistant (and also way faster). Furthermore, on this example, both reverse orderings are correct.
However, when making the new version the default one, many tests for is_chordal
and is_interval
are broken.
Any help is welcome.
David.
comment:23 Changed 4 years ago by
 Commit changed from 588c43140644c9d43cfae2e27e116566eda9ac54 to 12fdf755595a2a19a333e7928e8d1ef6d2e3f649
Branch pushed to git repo; I updated commit sha1. New commits:
12fdf75  Merge branch 'public/11736' in 7.5.b1

comment:24 Changed 4 years ago by
 Commit changed from 12fdf755595a2a19a333e7928e8d1ef6d2e3f649 to 906438689c7b8b32756c8799fb410682b0c1dea8
Branch pushed to git repo; I updated commit sha1. New commits:
9064386  trac 11736 pep8

comment:25 Changed 7 months ago by
 Branch changed from public/11736 to public/graphs/11736_lex_bfs
 Commit changed from 906438689c7b8b32756c8799fb410682b0c1dea8 to 89af4c2dc2f6a39f1d009c02feb945c00500cd28
 Milestone changed from sage6.4 to sage9.2
 Status changed from needs_info to needs_review
New branch based on 9.2.beta8 with:
 corrected (hope so) version of previous trials for implementing the algorithm of Habib et al 2000.
 several improvements of the existing code
Please test it extensively !
New commits:
89af4c2  trac #11736: new version

comment:26 Changed 7 months ago by
 Status changed from needs_review to needs_work
something wrong :(
File "src/sage/graphs/generic_graph.py", line 13620, in sage.graphs.generic_graph.GenericGraph.is_interval Failed example: [sum(1 for g in graphs(i) if g.is_interval()) for i in range(8)] # long time Expected: [1, 1, 2, 4, 10, 27, 92, 369] Got: [1, 1, 2, 4, 10, 27, 91, 374]
At least this graph is not recognized as interval graph (actually, not chordal) when using the new version: Graph("ETzw")
.
comment:27 Changed 7 months ago by
 Commit changed from 89af4c2dc2f6a39f1d009c02feb945c00500cd28 to 90d266e73f1f84d8fbae9b64a7ade43b459c80d6
Branch pushed to git repo; I updated commit sha1. New commits:
90d266e  trac #11736: fix memory allocation and add documentation

comment:28 Changed 7 months ago by
 Status changed from needs_work to needs_review
The error was a side effect of wrong memory allocation. Should be OK now.
comment:29 Changed 7 months ago by
 Cc ghkliem added; ncohen removed
comment:30 Changed 6 months ago by
 Cc mjo added
Is it feasible to generate a small random graph, feed it to both algorithms, and check that their outputs agree in a doctest? That would go a long way towards ensuring that the newer implementation is correct (or at least as correct as the old one).
comment:31 Changed 6 months ago by
The difficulty is that we may have several valid orderings, even with same initial vertex, and that the 2 algorithms may produce different orderings For instance:
sage: for i in range(100): ....: G = graphs.RandomChordalGraph(10) ....: F = G.lex_BFS(initial_vertex=0, algorithm="fast") ....: S = G.lex_BFS(initial_vertex=0, algorithm="slow") ....: print(i, "fast", F) ....: print(i, "slow", S) ....: if S != F: ....: print(G.graph6_string()) ....: break ....: 0 fast [0, 1, 5, 9, 6, 4, 7, 2, 8, 3] 0 slow [0, 1, 5, 9, 6, 4, 7, 2, 8, 3] 1 fast [0, 1, 4, 8, 2, 7, 9, 6, 3, 5] 1 slow [0, 1, 4, 8, 2, 7, 9, 6, 3, 5] 2 fast [0, 6, 7, 9, 8, 5, 1, 3, 2, 4] 2 slow [0, 6, 7, 9, 8, 1, 3, 5, 2, 4] IAHSCEA_w
In this example, the last graph is not connected, but it's the same with connected graphs. For instance, take a star with center 0. Then any permutation of the leaves lead to a valid lex BFS ordering.
One option could be to make an indirect doctest using is_chordal
on a random chordal graph, but for that me must be able to specify the lex BFS algorithm to use in is_chordal
. Not sure it's the best way.
But may be there is an other way to check that a lex BFS ordering is valid ?
comment:32 Changed 6 months ago by
Ok, I see the problem. The paper "LexBFSorderings and powers of graphs" (doi:10.1007/3540625593_15) has a characterization of LBFS orderings in Lemma 1 that looks pretty easy to implement. What do you think about an _is_lbfs(graph, order)
function that could be used to test the outputs of the fast/slow algorithms? Both would return True
when fed to _is_lbfs
, I expect.
comment:33 Changed 6 months ago by
Minor comment outside of the scope of the ticket: You might be able to do this for a bit more speed:
 def l_func(x):  return code[x] # The initial_vertex is at position 0 and so named 0 in sd code[0].append(n + 1) now = 1 while vertices:  v = max(vertices, key=l_func) + v = max(vertices, key=code.__getitem__)
comment:34 Changed 6 months ago by
 Commit changed from 90d266e73f1f84d8fbae9b64a7ade43b459c80d6 to 7609c669194ad947ad7203cb01f65f98d2c15f93
comment:35 Changed 6 months ago by
I have added a method to check that a lexBFS ordering is valid in (directed) graphs. I'm not yet 100% convinced that it is enough to check this property, so if you are aware of other tests, let me know.
comment:36 Changed 6 months ago by
We're working on getting rid of this, but for now, you have to call set_random_seed()
at the beginning of every doctest where randomness is required (i.e. the ones you've just added). Otherwise the random number generator isn't actually random, and you'll wind up testing the same example every time the tests are run.
comment:37 Changed 6 months ago by
Example:
The nonnegative orthant is a proper cone:: sage: set_random_seed() sage: ambient_dim = ZZ.random_element(10) sage: K = cones.nonnegative_orthant(ambient_dim) sage: K.is_proper() True
comment:38 Changed 6 months ago by
 Commit changed from 7609c669194ad947ad7203cb01f65f98d2c15f93 to 52c355e8d8aef1faee2ca5ed17da485e999279b9
Branch pushed to git repo; I updated commit sha1. New commits:
52c355e  trac #11736: set random seed

comment:39 Changed 6 months ago by
Thank you. I didn't know that.
comment:40 Changed 6 months ago by
I've been playing around with this, here are a few more suggestions:
 You could use
ZZ.random_element(G.num_verts())
instead of always0
for the initial vertex in your random tests; that would improve coverage a bit.
 The documentation for the
cdef
method isn't appearing in the reference manual for me (does anyone know if that's expected?). Since you put a lot of work into the docs forlex_BFS_fast_short_digraph
, maybe you want to move them into the mainlex_BFS
docstring?
 (Unrelated to this ticket) While you're in there, please add an extra
:
to the end ofTESTS:
inmaximum_cardinality_search()
. Right now sage doesn't recognize it as a doctest.
comment:41 Changed 6 months ago by
 Commit changed from 52c355e8d8aef1faee2ca5ed17da485e999279b9 to 15a041920a74a2268e98e787cf8da3200562ecc0
Branch pushed to git repo; I updated commit sha1. New commits:
15a0419  trac #11736: review comments

comment:42 Changed 6 months ago by
I have implemented your suggestions. For the algorithm, I made a copy of the description without suppressing it from the cdef method. I think it is easier this way for someone interested in this code to understand it.
comment:43 Changed 6 months ago by
Thanks! Here are two more small things:
First, in the documentation for lex_BFS_fast_short_digraph
(and now lex_BFS
), there's a LaTeX typo at
S = `{u  i \leq \sigma(u) \leq j\}
The opening curly brace isn't escaped, so the formula isn't showing up.
Second, in _is_valid_lex_BFS_order
, it looks to me like you've figured out the correct directions for the edges to make things work with a digraph, namely ca
, cb
, da
, and db
. However the docs still have some of them in a different order from when the result was stated for an undirected graph:
if `a < b < c` and `ac \in E` and `bc \not\in E`, then there exists `d\in V` such that `c < d`, `bd \in E` and `ad \not\in E`.
comment:44 Changed 6 months ago by
 Commit changed from 15a041920a74a2268e98e787cf8da3200562ecc0 to 332fb8bf3acb23d8cb1a376744665cd25c680c26
Branch pushed to git repo; I updated commit sha1. New commits:
332fb8b  trac #11736: fix documentation

comment:45 Changed 6 months ago by
good catch. Hope it will be ok this time.
comment:46 Changed 6 months ago by
 Reviewers set to Michael Orlitzky
 Status changed from needs_review to positive_review
This looks good to me now, unless anyone else wants to comment. I've checked the implementation of _is_valid_lex_BFS_order
. It's "obviously correct" for undirected graphs, and probably for directed graphs too although I haven't been able to find the corresponding theorem for digraphs written down.
The implementation of lex_BFS_fast_short_digraph
is less scrutable but looks reasonable now that I understand how the algorithm works. I trust the doctests more than I trust myself in any case. Since you've been working on this for 8 years I'm inclined to say let's get it over with.
comment:47 Changed 6 months ago by
Thank you for your help.
To understand the fast algorithm, you need to understand that a slice is nothing else than a label. When creating a subslice of a slice from position p to position q, you place the vertices of the sub slice on positions p...p+I, and the remaining vertices of the original slice at positions p+I+1...q. So the vertices in the sub slice have a larger label than the other vertices of the original slice. The tricks are to associate a position with the current slice name, to maintain the head (left most vertex of a slice) of each slice, and give a name to each newly created subslice.
To check the property on directed graphs, its more tricky, but I'm convinced it is correct. An alternative to check the validity is to modify the slow algorith. That is, instead of selecting a vertex with largest label, we identify all candidates (vertices with same largest label), and check that the next vertex in the given order is in this pool of candidates, then select it, update labels of its neighbors and continue. We can add that if needed in a future ticket.
comment:48 Changed 6 months ago by
 Description modified (diff)
comment:49 Changed 6 months ago by
 Branch changed from public/graphs/11736_lex_bfs to 332fb8bf3acb23d8cb1a376744665cd25c680c26
 Resolution set to fixed
 Status changed from positive_review to closed
Hello Diego !!!
Same comment (see #11738) for this patch about the "build/" folder. I also expect that this patch breaks doctests in Sage's library, so please check it too, correcting the doctests themselves if it makes sense
:)
Nathann