tag:blogger.com,1999:blog-1700157236206200597.post3308933040690383564..comments2011-04-21T10:02:54.679-07:00Peter Thatcherhttp://www.blogger.com/profile/01092342988993218446noreply@blogger.comBlogger13125tag:blogger.com,1999:blog-1700157236206200597.post-61631768747119289402011-04-21T10:02:54.679-07:002011-04-21T10:02:54.679-07:00ah this is how:<br /><br /> value, state = expr.run(ParserState(ByteStream(inputline)))<br /> if state.position != len(inputline):<br /> raise ParseError(&quot;couldn&#39;t parse entire line %s, left over: %s&quot; % (<br /> inputline, inputline[state.position:]))Laxori666http://www.blogger.com/profile/03434854378035632469noreply@blogger.comtag:blogger.com,1999:blog-1700157236206200597.post-38006434025233985352011-04-21T09:30:13.688-07:002011-04-21T09:30:13.688-07:00how do you tell whether the whole input stream was parsed?Laxori666http://www.blogger.com/profile/03434854378035632469noreply@blogger.comtag:blogger.com,1999:blog-1700157236206200597.post-26927164106368957942009-01-18T15:40:00.000-08:002009-01-18T15:40:00.000-08:00Hi Peter - <BR/> Great! Thanks very much for your reply! <BR/> Pysec is a great little app when you're getting into Python and are also keen on FP (as I am). <BR/> As a token of thanks, here's a link to another *very cool* Python parsing app. It's called "yeanpypa" (YEt ANother PYthon PArser lib). Very nicely done - it is inspired by the C++ Boost::Spirit parser library (which I have a fair bit of experience with, and which rocks!).<BR/>Yeanpypa doesn't use an FP approach, but it's quite a nice "BNF"-y style parser. <BR/>http://freshmeat.net/projects/yeanpypa/Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-1700157236206200597.post-79137569756311588722009-01-07T05:20:00.000-08:002009-01-07T05:20:00.000-08:00Andy, <BR/><BR/>I'm glad you like it. Unfortunately, for my main use, Pysec too slow (I made a blog entry about this called "why are my monads so slow?"), and so I haven't worked on it at all. <BR/><BR/>Hopefully it will be fast enough and work well enough for your needs. You're free to do whatever you want with it. Consider it "public domain", I guess.Peter Thatcherhttp://www.blogger.com/profile/01092342988993218446noreply@blogger.comtag:blogger.com,1999:blog-1700157236206200597.post-37653547950996048552008-12-22T17:31:00.000-08:002008-12-22T17:31:00.000-08:00Hi Peter! <BR/><BR/>I've just come across Pysec - a great parser library, I love it! <BR/><BR/>I've known about Haskell's Parsec for some time (and that's very impressive too), but it is good to see a similar library in a somewhat-more-familiar language. Haskell has a bit of a learning-curve to it.... ;) <BR/><BR/>I'm very keen to use this library in the near-future. I was wondering about the license - is it "public domain"? ( I always give credit anyway... ) <BR/> Many thanks for this post (and for a great library! <BR/> - AndyAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-1700157236206200597.post-72663050597066490962008-02-14T08:40:00.000-08:002008-02-14T08:40:00.000-08:00Perhaps I can characterize Pysec/Parsec as a way to build parsers which are macro-expansion-friendly. <BR/>I think the general case of this is that they allow parsers to change behavior based on what as already been parsed, which, in a way, is viewing the parsed text less as a language of a grammar and more as processing instructions.Peter Thatcherhttp://www.blogger.com/profile/01092342988993218446noreply@blogger.comtag:blogger.com,1999:blog-1700157236206200597.post-12451988092788954692008-02-14T05:33:00.000-08:002008-02-14T05:33:00.000-08:00Don't be too impressed. The only reason I was able to crank this out so quickly was because it was mostly a replication of a helper function already defined in pyparsing, called countedArray. Now that you've see the other implementation, look at the implementation of countedArray (here: http://pyparsing.pastebin.com/m278ec49a). It also uses a "global" value (the variable arrayExpr), but at least it hides it inside the countedArray method.<BR/><BR/>My experience with macro expansion is that it is not a parser-friendly process in general. For instance, unless your parser engine allows you to back up to reparse generated text, you have to perform multiple passes on the input source - one to do macro expansion (potentially recursively, which should still be doable in a single pass), and then a second pass to parse the macro-expanded code. I don't really feel too bad about this, C compilers have to do much the same thing.<BR/><BR/>-- PaulPaul McGuirehttp://www.geocities.com/ptmcgnoreply@blogger.comtag:blogger.com,1999:blog-1700157236206200597.post-8106846850165475142008-02-14T03:23:00.000-08:002008-02-14T03:23:00.000-08:00Paul, thanks for your response. It helped me compare pyparsing and Pysec. I'll have to write another article comparing the two, but here's what I found initially:<BR/><BR/>1. It can parse b7:XXXXXXX, but it's a hack. I'm impressed you figured out such a hack so quickly, but it essentially requires modifying a global variable to work. <BR/><BR/>2. It can parse [&a [1, 2, 3] @a @a]. A tweaking of the macro expansion on the wiki is sufficient. But I'm calling this one a hack too because it also requires modifying a global value (the macro table). <BR/><BR/>So, it works for my purposes, but I don't really like how. Admittedly, the format I'm parsing isn't definable as a grammar, and so a parsing library built around the idea of a grammar doesn't match too well.Peter Thatcherhttp://www.blogger.com/profile/01092342988993218446noreply@blogger.comtag:blogger.com,1999:blog-1700157236206200597.post-42558262558786684672008-02-13T21:03:00.000-08:002008-02-13T21:03:00.000-08:00Peter -<BR/><BR/>For an implementation of the size-prefixed binary data, see: <BR/>http://pyparsing.pastebin.com/m72db9ab6<BR/><BR/>As for the second question, I don't quite understand what you are describing, but there is a macroProcessor example on the pyparsing wiki Examples page.<BR/><BR/>Pyparsing has epydoc-generated class docs, UML diagrams, and two presentations I made at Pycon06, that are included with the source or docs release on SourceForge. If you go to the Documentation page of the wiki, there are some links to articles, blog entries, and if you want to spring for $10, a ShortCut on O'Reilly.<BR/><BR/>Post a comment on the wiki home page Discussion tab, if you want to pursue item #2 further.<BR/><BR/>-- PaulPaul McGuirehttp://pyparsing.wikispaces.comnoreply@blogger.comtag:blogger.com,1999:blog-1700157236206200597.post-24014476636240796212008-02-13T15:27:00.000-08:002008-02-13T15:27:00.000-08:00Paul, I had no idea pyparsing even existed. I hate reinventing the wheel and searched far and wide for parsing tools, but pyparsing never came up. Now when I Google for "python parsing", I see that pyparsing is the last result on the first page, but I guess my eyes missed it when I looked before. Shame on me.<BR/><BR/>I congratulate you on your work. It looks great. It's almost what I was trying to accomplish, but certainly more mature. Had I known about it, I may have used it instead, although I'm glad that I could explore monadic parsing as well.<BR/><BR/>But, I have some peculiar needs in a parser that I wrote using Pysec. I'd like to see if pyparsing can do it also, but there's not much documentation that I can find on the wiki. Would you mind answering a few questions? You can respond here or direct me to a better place to have a discussion.<BR/><BR/>Here's what I'm wondering if pyparsing is capable of:<BR/><BR/>1. I need to read size-prefixed binary data, much like what's found in the bittorrent format called bencode. It looks like this:<BR/><BR/>b7:XXXXXXX<BR/><BR/>Where each X is a byte. The parser has to read the 7, parse it, and then read 7 bytes, no matter what they are, and then continue parsing from there. Can pyparsing do this? In Pysec, it looks a bit like this:<BR/><BR/>(match("b") & integer) >> (lambda size : (match(":") & read(size))) <BR/><BR/>2. I need something a lot like macro expansion, except that rather then expand the reference into the text of the anchor, I need it to expand into the parsed value of the anchor. So, for example, I need to parse something that looks like this:<BR/><BR/>[&a [1, 2, 3], @a, @a]<BR/><BR/>Into what you would get if you did this in python:<BR/><BR/>a = [1, 2, 3]<BR/>[a, a, a]<BR/><BR/>It requires keeping a table of parsed values, but that's easy because a Pysec Parser is really a StateTransformer monad, so we just carry an inner state in around with the ParserState.<BR/><BR/>Can pyparsing do this?<BR/><BR/>Thanks in advance Paul.Peter Thatcherhttp://www.blogger.com/profile/01092342988993218446noreply@blogger.comtag:blogger.com,1999:blog-1700157236206200597.post-18013952645959451542008-02-13T13:50:00.000-08:002008-02-13T13:50:00.000-08:00Congratulations on a nice little parsing library. For comparison, here is a pyparsing JSON parser (http://pyparsing.wikispaces.com/space/showimage/jsonParser.py). It should look very familiar to you! :)<BR/><BR/>Come visit the pyparsing wiki, the concepts are very similar. One of the big differences is that your monads are implemented in functions, and pyparsing uses class instances. By using objects, it is easier to attach additional behavior (such as parse actions and field names) to the parser expression monads. Also, pyparsing creates a very rich return type, the ParseResults class, instead of just returning nested lists of tokens.Paul McGuirehttp://pyparsing.wikispaces.comnoreply@blogger.comtag:blogger.com,1999:blog-1700157236206200597.post-25411889564779507902008-02-12T21:04:00.000-08:002008-02-12T21:04:00.000-08:00How's this different from pyparsing?Allen Shorthttp://www.blogger.com/profile/02476178041186364808noreply@blogger.comtag:blogger.com,1999:blog-1700157236206200597.post-58359358130276128822008-02-12T15:56:00.000-08:002008-02-12T15:56:00.000-08:00that's really good!! lets kill regular expression :)jbhttp://www.blogger.com/profile/13709249214033017810noreply@blogger.com