[PEAK] trying to run peak-rules on python3

[Hervé Coatanhay]

Hi,

I'm trying to port nagare to python3. I managed to run a simple application
but I had to make some changes in few PEAK packages.

Here is how I proceeded:
1) run 2to3 on these packages
2) fix remaining "obvious" issue (no more `file` builtin, package renaming,
...)
3) fix types/new/ast changes made in python3
4) remove rules on old-style classes

Here is the code produced by this very naive approach
https://bitbucket.org/Alzakath/peak-py3.

As a result, PEAK-Rules seems to work when I only use dispatch on type. But
it seems predicate dispatch is not working, for example:

from peak.rules import when

def f(x):
print('default')

@when(f, 'x == 3')
def f(x):
print("I'm an integer")

f('test')
f(3)


Traceback (most recent call last):
File "peak-rules_test_ko.py", line 6, in <module>
@when(f, 'x == 3')
File
"/Users/herve/Documents/workspace/nagare/python3-ports/peak-py3/DecoratorTools-1.8/peak/util/decorators.py",
line 644, in do_decorate
frame, getattr(f,'__name__',None), f, frame.f_locals
File
"/Users/herve/Documents/workspace/nagare/python3-ports/peak-py3/PEAK-Rules-0.5a1.dev-r2713/peak/rules/core.py",
line 278, in callback
register_for_class(None)
File
"/Users/herve/Documents/workspace/nagare/python3-ports/peak-py3/PEAK-Rules-0.5a1.dev-r2713/peak/rules/core.py",
line 272, in register_for_class
_register_rule(f, pred, context, cls)
File
"/Users/herve/Documents/workspace/nagare/python3-ports/peak-py3/PEAK-Rules-0.5a1.dev-r2713/peak/rules/core.py",
line 398, in _register_rule
rules.add(parse_rule(Dispatching(gf).engine, pred, context, cls))
File
"/Users/herve/Documents/workspace/nagare/python3-ports/peak-py3/PEAK-Rules-0.5a1.dev-r2713/peak/rules/core.py",
line 689, in parse_rule
def parse_rule(engine, predicate, context, cls):
File
"/Users/herve/Documents/workspace/nagare/python3-ports/peak-py3/PEAK-Rules-0.5a1.dev-r2713/peak/rules/core.py",
line 592, in callback
return f(*args, **kw)
File
"/Users/herve/Documents/workspace/nagare/python3-ports/peak-py3/PEAK-Rules-0.5a1.dev-r2713/peak/rules/core.py",
line 952, in parse_upgrade
from peak.rules.predicates import IndexedEngine
File
"/Users/herve/Documents/workspace/nagare/python3-ports/peak-py3/PEAK-Rules-0.5a1.dev-r2713/peak/rules/predicates.py",
line 4, in <module>
from .criteria import *
File
"/Users/herve/Documents/workspace/nagare/python3-ports/peak-py3/PEAK-Rules-0.5a1.dev-r2713/peak/rules/criteria.py",
line 145, in <module>
inequalities = {
TypeError: nonempty __slots__ not supported for subtype of 'int'

Does anyone tried to port peak-rules to python3 ? Do you have any advice on
the right way to do it ? Or even hints on what I'm missing ?

Cheers,
Herve
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.eby-sarna.com/pipermail/peak/attachments/20130318/c215de4d/attachment.html

[PJ Eby]

On Mon, Mar 18, 2013 at 1:56 PM, Herv? Coatanhay

Post by Hervé Coatanhay
"/Users/herve/Documents/workspace/nagare/python3-ports/peak-py3/PEAK-Rules-0.5a1.dev-r2713/peak/rules/criteria.py",
line 145, in <module>
inequalities = {
TypeError: nonempty __slots__ not supported for subtype of 'int'
Does anyone tried to port peak-rules to python3 ? Do you have any advice on
the right way to do it ? Or even hints on what I'm missing ?

Well, the nonempty slots thing is probably about the
peak.rules.criteria.IsObject class, which would need some refactoring
to not be based on the integer type. I couldn't tell you how many
other things might not work; for example, did you get
BytecodeAssembler to pass all its test? For that matter, which of
PEAK-Rules' own tests are not working?

[Hervé Coatanhay]

Ok I totally missed the tests in README.txt.
I have to make them python3 friendly before running them.
I'll come up with something more consistent as soon as it is done.

Post by PJ Eby
On Mon, Mar 18, 2013 at 1:56 PM, Herv? Coatanhay
"/Users/herve/Documents/workspace/nagare/python3-ports/peak-py3/PEAK-Rules-0.5a1.dev-r2713/peak/rules/criteria.py",

Post by Hervé Coatanhay
line 145, in <module>
inequalities = {
TypeError: nonempty __slots__ not supported for subtype of 'int'
Does anyone tried to port peak-rules to python3 ? Do you have any advice

on

Post by Hervé Coatanhay
the right way to do it ? Or even hints on what I'm missing ?

Well, the nonempty slots thing is probably about the
peak.rules.criteria.IsObject class, which would need some refactoring
to not be based on the integer type. I couldn't tell you how many
other things might not work; for example, did you get
BytecodeAssembler to pass all its test? For that matter, which of
PEAK-Rules' own tests are not working?

[Hervé Coatanhay]

Skipped content of type multipart/alternative-------------- next part --------------
F.
======================================================================
FAIL: Doctest: README.txt
----------------------------------------------------------------------
Traceback (most recent call last):
File "/opt/stackless-3.2.2/lib/python3.2/doctest.py", line 2118, in runTest
raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for README.txt
File "/Users/herve/Documents/workspace/nagare/python3-ports/peak-py3/BytecodeAssembler-0.6/README.txt", line 0

----------------------------------------------------------------------
File "/Users/herve/Documents/workspace/nagare/python3-ports/peak-py3/BytecodeAssembler-0.6/README.txt", line 498, in README.txt
Failed example:
dis(c.code())
Expected:
0 0 LOAD_CONST 1 (1)
3 LOAD_CONST 2 (True)
6 LOAD_CONST 3 (1.0)
9 LOAD_CONST 4 (1)
Got:
0 0 LOAD_CONST 1 (1)
3 LOAD_CONST 2 (True)
6 LOAD_CONST 3 (1.0)
9 LOAD_CONST 1 (1)
----------------------------------------------------------------------
File "/Users/herve/Documents/workspace/nagare/python3-ports/peak-py3/BytecodeAssembler-0.6/README.txt", line 1457, in README.txt
Failed example:
dis(f4)
Expected:
0 0 LOAD_FAST 1 (.1)
3 UNPACK_SEQUENCE 2
6 STORE_FAST 3 (b)
9 STORE_FAST 4 (c)
12 LOAD_FAST 2 (.2)
15 UNPACK_SEQUENCE 2
18 STORE_FAST 5 (d)
21 UNPACK_SEQUENCE 2
24 STORE_FAST 6 (e)
27 STORE_FAST 7 (f)
Got nothing
----------------------------------------------------------------------
File "/Users/herve/Documents/workspace/nagare/python3-ports/peak-py3/BytecodeAssembler-0.6/README.txt", line 1869, in README.txt
Failed example:
dump(c.code())
Expected:
SETUP_EXCEPT L1
LOAD_CONST 1 (1)
RETURN_VALUE
POP_BLOCK
JUMP_FORWARD L4
L1: DUP_TOP
LOAD_CONST 2 (<class 'KeyError'>)
COMPARE_OP 10 (exception match)
JUMP_IF_FALSE L2
POP_TOP
POP_TOP
POP_TOP
POP_TOP
LOAD_CONST 3 (2)
JUMP_FORWARD L5
L2: POP_TOP
DUP_TOP
LOAD_CONST 4 (<class 'KeyError'>)
COMPARE_OP 10 (exception match)
JUMP_IF_FALSE L3
POP_TOP
POP_TOP
POP_TOP
POP_TOP
LOAD_CONST 5 (3)
JUMP_FORWARD L5
L3: POP_TOP
END_FINALLY
L4: LOAD_CONST 6 (4)
RETURN_VALUE
L5: RETURN_VALUE
Got:
SETUP_EXCEPT L1
LOAD_CONST 1 (1)
RETURN_VALUE
POP_BLOCK
JUMP_FORWARD L4
L1: DUP_TOP
LOAD_CONST 2 (<class 'KeyError'>)
COMPARE_OP 10 (exception match)
JUMP_IF_FALSE L2
POP_TOP
POP_TOP
POP_TOP
POP_TOP
LOAD_CONST 3 (2)
JUMP_FORWARD L5
L2: POP_TOP
DUP_TOP
LOAD_CONST 4 (<class 'TypeError'>)
COMPARE_OP 10 (exception match)
JUMP_IF_FALSE L3
POP_TOP
POP_TOP
POP_TOP
POP_TOP
LOAD_CONST 5 (3)
JUMP_FORWARD L5
L3: POP_TOP
END_FINALLY
L4: LOAD_CONST 6 (4)
RETURN_VALUE
L5: RETURN_VALUE
----------------------------------------------------------------------
File "/Users/herve/Documents/workspace/nagare/python3-ports/peak-py3/BytecodeAssembler-0.6/README.txt", line 2850, in README.txt
Failed example:
dis(c.code())
Expected:
0 0 LOAD_CONST 1 (1)
3 LOAD_CONST 2 (2)
6 BUILD_TUPLE 2
9 UNPACK_SEQUENCE 2
12 DUP_TOPX 2
15 LOAD_CONST 3 (3)
18 BUILD_LIST 5
21 UNPACK_SEQUENCE 5
24 BUILD_SLICE 3
27 BUILD_SLICE 3
Got:
0 0 LOAD_CONST 1 (1)
3 LOAD_CONST 2 (2)
6 BUILD_TUPLE 2
9 UNPACK_SEQUENCE 2
12 DUP_TOP_TWO
13 ROT_TWO
14 STOP_CODE
15 LOAD_CONST 3 (3)
18 BUILD_LIST 5
21 UNPACK_SEQUENCE 5
24 BUILD_SLICE 3
27 BUILD_SLICE 3
----------------------------------------------------------------------
File "/Users/herve/Documents/workspace/nagare/python3-ports/peak-py3/BytecodeAssembler-0.6/README.txt", line 2952, in README.txt
Failed example:
dis(c.code())
Expected:
1 0 LOAD_FAST 0 (.0)
3 UNPACK_SEQUENCE 2
6 STORE_FAST 1 (x)
9 STORE_FAST 2 (y)
Got nothing
----------------------------------------------------------------------
File "/Users/herve/Documents/workspace/nagare/python3-ports/peak-py3/BytecodeAssembler-0.6/README.txt", line 2960, in README.txt
Failed example:
dis(c.code())
Expected:
1 0 LOAD_FAST 1 (.1)
3 UNPACK_SEQUENCE 2
6 STORE_FAST 2 (y)
9 UNPACK_SEQUENCE 3
12 STORE_FAST 3 (z)
15 STORE_FAST 4 (a)
18 STORE_FAST 5 (b)
Got nothing


----------------------------------------------------------------------
Ran 2 tests in 0.360s

FAILED (failures=1)

[PJ Eby]

On Tue, Mar 19, 2013 at 11:04 AM, Herv? Coatanhay

FYI
I did changes in README.txt to pass test. Mostly obvious ones, like print
and types changes. But i had to replace DUP_TOPX with DUP_TOP_TWO (I'm not
entirely sure if that's OK).

Me either. I haven't messed with py3 bytecode at all yet. But if I
understand correctly, DUP_TOPX(1) would map to DUP_TOP and DUP_TOPX(2)
would map to DUP_TOP_TWO, but any larger number and it's more
complicated than that.

There are still some failing tests. One seems like some discrepancy between

exec(compile('42', '', 'eval'))
eval(compile('42', '', 'eval'))

42

exec(compile('42', '', 'eval'))

42

eval(compile('42', '', 'eval'))

42

That one probably won't affect anything in PEAK-Rules.

Other failing tests a related to the use of from_function classmethod. I'm
suspecting some issue with DecoratorTools because Decoratools tests are not
passing. I'll try to make them pass.

from peak.util.assembler import TryExcept
c = Code()
c.return_(

... TryExcept(
... Return(1), # body
... [(Const(KeyError),2), (Const(TypeError),3)], # handlers
... Return(4) # else
clause
... )
... )

Not sure what's up with this one: the log you sent shows this actually
having the same code (or at least as far as I can see). It shouldn't
be a whitespace difference, as I think the flags are set to ignore
those.

c = Code()
c(1, True, 1.0, 1) # equal, but different types
dis(c.code())

This is a 2to3 artifact: the original test had the fourth value as
"1L", which resulted in a fourth constant. Python 3 doesn't have a
distinct type for longs, so there's no way to create that value. So
this failure is of no consequence.

Anyway, what about PEAK-Rules' own tests? It's got quite a lot of
them, including for the criteria and predicate and indexing systems.

[Hervé Coatanhay]

So from what I'm understanding BytesCodeAssembler is not in a too bad shape
atm regarding py3 if i just want to run PEAK-Rules (i still have to look
into the problem with `from_function`).

The way i run the test is:
nosetests --with-doctest --doctest-extension=txt
--doctest-options='+ELLIPSIS,+NORMALIZE_WHITESPACE'

so whitespaces differences should be ignored properly

from peak.util.assembler import TryExcept
c = Code()
c.return_(

... TryExcept(
... Return(1), # body
... [(Const(KeyError),2), (Const(TypeError),3)], # handlers
... Return(4) # else
clause
... )
... )

It was a typo as i replace both exceptions.KeyError and
exceptions.TypeError with KeyError.

As of running PEAK-Rules tests, i have trouble running them because the
previous command does not find any. How do you run them actually ?

Herve

On Tue, Mar 19, 2013 at 11:04 AM, Herv? Coatanhay

FYI
I did changes in README.txt to pass test. Mostly obvious ones, like print
and types changes. But i had to replace DUP_TOPX with DUP_TOP_TWO (I'm

not

entirely sure if that's OK).

Me either. I haven't messed with py3 bytecode at all yet. But if I
understand correctly, DUP_TOPX(1) would map to DUP_TOP and DUP_TOPX(2)
would map to DUP_TOP_TWO, but any larger number and it's more
complicated than that.

There are still some failing tests. One seems like some discrepancy

between

exec(compile('42', '', 'eval'))
eval(compile('42', '', 'eval'))

42

exec(compile('42', '', 'eval'))

42

eval(compile('42', '', 'eval'))

42

That one probably won't affect anything in PEAK-Rules.

Other failing tests a related to the use of from_function classmethod.

I'm

suspecting some issue with DecoratorTools because Decoratools tests are

not

passing. I'll try to make them pass.

from peak.util.assembler import TryExcept
c = Code()
c.return_(

... TryExcept(
... Return(1), # body
... [(Const(KeyError),2), (Const(TypeError),3)], #

handlers

... Return(4) # else
clause
... )
... )

Not sure what's up with this one: the log you sent shows this actually
having the same code (or at least as far as I can see). It shouldn't
be a whitespace difference, as I think the flags are set to ignore
those.

c = Code()
c(1, True, 1.0, 1) # equal, but different types
dis(c.code())

This is a 2to3 artifact: the original test had the fourth value as
"1L", which resulted in a fourth constant. Python 3 doesn't have a
distinct type for longs, so there's no way to create that value. So
this failure is of no consequence.
Anyway, what about PEAK-Rules' own tests? It's got quite a lot of
them, including for the criteria and predicate and indexing systems.

[PJ Eby]

On Tue, Mar 19, 2013 at 2:57 PM, Herv? Coatanhay

Post by Hervé Coatanhay
So from what I'm understanding BytesCodeAssembler is not in a too bad shape
atm regarding py3 if i just want to run PEAK-Rules (i still have to look
into the problem with `from_function`).

PEAK-Rules does depend on from_function, so yes.

Most of the problems in the tests that you sent earlier had to do with
nested arguments, e.g "def x((y,z), a):", which are not supported on
Python 3. So it may be that it's in that functionality where you're
experiencing issues.

Post by Hervé Coatanhay
nosetests --with-doctest --doctest-extension=txt
--doctest-options='+ELLIPSIS,+NORMALIZE_WHITESPACE'
so whitespaces differences should be ignored properly

from peak.util.assembler import TryExcept
c = Code()
c.return_(

... TryExcept(
... Return(1), # body
... [(Const(KeyError),2), (Const(TypeError),3)], # handlers
... Return(4) # else
clause
... )
... )

It was a typo as i replace both exceptions.KeyError and exceptions.TypeError
with KeyError.
As of running PEAK-Rules tests, i have trouble running them because the
previous command does not find any. How do you run them actually ?

For both packages, you should be using "python setup.py test", or you
will miss stuff or have incorrect option settings.

[Hervé Coatanhay]

Skipped content of type multipart/alternative-------------- next part --------------
running test
running egg_info
writing requirements to PEAK_Rules.egg-info/requires.txt
writing PEAK_Rules.egg-info/PKG-INFO
writing namespace_packages to PEAK_Rules.egg-info/namespace_packages.txt
writing top-level names to PEAK_Rules.egg-info/top_level.txt
writing dependency_links to PEAK_Rules.egg-info/dependency_links.txt
reading manifest file 'PEAK_Rules.egg-info/SOURCES.txt'
writing manifest file 'PEAK_Rules.egg-info/SOURCES.txt'
running build_ext
test_basics (test_rules.DecompilationTests) ... ERROR
test_calls (test_rules.DecompilationTests) ... ERROR
test_dicts (test_rules.DecompilationTests) ... ERROR
test_paren_precedence (test_rules.DecompilationTests) ... ERROR
test_powers (test_rules.DecompilationTests) ... ERROR
test_slices (test_rules.DecompilationTests) ... ERROR
testAbstract (test_rules.MiscTests) ... ok
testAbstractRegeneration (test_rules.MiscTests) ... /Users/herve/Documents/workspace/nagare/python3-ports/peak-py3/PEAK-Rules-0.5a1.dev-r2713/peak/rules/core.py:139: DeprecationWarning: tostring() is deprecated. Use tobytes() instead.
self.backup = c.code()
ok
testClassBodyRules (test_rules.MiscTests) ... /Users/herve/Documents/workspace/nagare/python3-ports/peak-py3/PEAK-Rules-0.5a1.dev-r2713/peak/rules/codegen.py:216: DeprecationWarning: tostring() is deprecated. Use tobytes() instead.
func.__code__ = self.code.code()
ERROR
testCreateEngine (test_rules.MiscTests) ... /Users/herve/Documents/workspace/nagare/python3-ports/peak-py3/PEAK-Rules-0.5a1.dev-r2713/test_rules.py:120: DeprecationWarning: Please use assertTrue instead.
self.failUnless(isinstance(new_engine, MyEngine))
ok
testEngineArgnames (test_rules.MiscTests) ... ok
testFlatPriorities (test_rules.MiscTests) ... /Users/herve/Documents/workspace/nagare/python3-ports/peak-py3/BytecodeAssembler-0.6/peak/util/assembler.py:1231: DeprecationWarning: tostring() is deprecated. Use tobytes() instead.
return Const(eval(c.code()))
ERROR
testIndexClassicMRO (test_rules.MiscTests) ... ERROR
testIndexedEngine (test_rules.MiscTests) ... ERROR
testInheritance (test_rules.MiscTests) ... ERROR
testIsNot (test_rules.MiscTests) ... ERROR
testNamedGFExtension (test_rules.MiscTests) ... ERROR
testNotInherited (test_rules.MiscTests) ... ERROR
testParseInequalities (test_rules.MiscTests) ... ERROR
testPointers (test_rules.MiscTests) ... ERROR
testRuleSetClear (test_rules.MiscTests) ... ok
testRuleSetReentrance (test_rules.MiscTests) ... ok
testSignatureOfTruthTests (test_rules.MiscTests) ... ok
testTypeEngineKeywords (test_rules.MiscTests) ... ok
testTypeImplicationAndIsSubclassOrdering (test_rules.MiscTests) ... ERROR
testTypeVsIsTypePrecedence (test_rules.MiscTests) ... ok
test_istype_implies (test_rules.MiscTests) ... /Users/herve/Documents/workspace/nagare/python3-ports/peak-py3/PEAK-Rules-0.5a1.dev-r2713/test_rules.py:160: DeprecationWarning: Please use assertFalse instead.
self.failIf(implies(istype(object, False), object))
ok
testClassNode (test_rules.NodeBuildingTests) ... ERROR
testIdentityNode (test_rules.NodeBuildingTests) ... ok
testRangeNode (test_rules.NodeBuildingTests) ... ERROR
testTruthNode (test_rules.NodeBuildingTests) ... ok
testKwArgHandling (test_rules.RuleDispatchTests) ... ERROR
testSimplePreds (test_rules.RuleDispatchTests) ... ERROR
testSingles (test_rules.RuleDispatchTests) ... ok
testSubclassDispatch (test_rules.RuleDispatchTests) ... ERROR
testTrivialities (test_rules.RuleDispatchTests) ... ok
testVarArgHandling (test_rules.RuleDispatchTests) ... ERROR
test_NoApplicableMethods_is_raised (test_rules.RuleDispatchTests) ... ERROR
testAroundDecoratorAndRetroactiveCombining (test_rules.TypeEngineTests) ... ok
testIntraSignatureCombinationAndRemoval (test_rules.TypeEngineTests) ... ok
/Users/herve/Documents/workspace/nagare/python3-ports/peak-py3/PEAK-Rules-0.5a1.dev-r2713/README.txt
Doctest: README.txt ... FAIL
/Users/herve/Documents/workspace/nagare/python3-ports/peak-py3/PEAK-Rules-0.5a1.dev-r2713/DESIGN.txt
Doctest: DESIGN.txt ... FAIL
/Users/herve/Documents/workspace/nagare/python3-ports/peak-py3/PEAK-Rules-0.5a1.dev-r2713/Indexing.txt
Doctest: Indexing.txt ... FAIL
/Users/herve/Documents/workspace/nagare/python3-ports/peak-py3/PEAK-Rules-0.5a1.dev-r2713/AST-Builder.txt
Doctest: AST-Builder.txt ... FAIL
/Users/herve/Documents/workspace/nagare/python3-ports/peak-py3/PEAK-Rules-0.5a1.dev-r2713/Code-Generation.txt
Doctest: Code-Generation.txt ... FAIL
/Users/herve/Documents/workspace/nagare/python3-ports/peak-py3/PEAK-Rules-0.5a1.dev-r2713/Syntax-Matching.txt
Doctest: Syntax-Matching.txt ... FAIL
/Users/herve/Documents/workspace/nagare/python3-ports/peak-py3/PEAK-Rules-0.5a1.dev-r2713/Criteria.txt
Doctest: Criteria.txt ... FAIL
/Users/herve/Documents/workspace/nagare/python3-ports/peak-py3/PEAK-Rules-0.5a1.dev-r2713/Predicates.txt
Doctest: Predicates.txt ...

[PJ Eby]

On Wed, Mar 20, 2013 at 10:54 AM, Herv? Coatanhay

I ran test for PEAK-Rules (see attached file) at the end there is a
"Segmentation fault: 11".
I'm leaving PyCon today and heading back to France so I won't be able to
look into it until this week-end.

I can't tell much from that file; it seems to be lacking the stderr
output (which contains all the actual errors, failures, etc.).

[PJ Eby]

On Wed, Mar 27, 2013 at 9:07 AM, Herv? Coatanhay

Ok so I had to remove Predicates.txt form the list of files in
test_rules.py.

Looks like you've got a lot of work ahead of you. Lots of stuff is
shallow failures due to reprs, tests not being in Python 3 syntax
(e.g. "print" statements), sorting issues in Python 3 (you can't
compare None to other types), lack of "long" type, set repr being {}
instead of ([]) as in Python 2, etc.

The good news is that a few small changes will fix a lot of errors in
that log. The bad news is that you must make the changes first,
because these shallow problems are preventing the tests from showing
us what the *real* problems are.

I did however spot one thing that looks like a real problem: it
appears that BytecodeAssembler's use of .tostring() calls should
change to .tobytes(), because the __code__ attribute in Python 3 is a
bytestring. I think this is causing most of the errors in
Code-Generation.txt.

A few other things I found:

* It appears the codegen module isn't fully importing; it should
export a Repr, but doesn't. It'd be good to get the actual error
message, if any.
* It appears that certain SLICE_* opcodes have gone away in Python 3;
don't know what they're replaced with.
* Don't bother trying to fix the README and DESIGN errors, or the ones
in test_rules itself; they are all being caused by a lower-level
failure, probably in the peak.rules.criteria, .indexing, .codegen, or
.predicates modules, if not BytecodeAssembler itself.

Good luck. ;-)

(Btw, It would probably be a good idea to send the next log to a paste
website rather than attaching it, given the size of these things.)