[WIP] v1.0.0 proposal

Drop support for byRefToken in ArgumentExpression - it's a syntax error
in php 7.0+ and is prohibited in php 5.4+.

Convert the foo and otherFooList to fooList for union types in catch
name lists, parameters, properties, and return union types.

Remove ThrowStatement in favor of ThrowExpression for php 8.0.
Remove EchoExpression in favor of EchoStatement - echo cannot be used as
an expression.

Remove statement from NamedLabelStatement - it's always null since a
named label is a single statement by itself.

TODO: Consistently support MissingToken in a single-element list in cases where
a non-empty list is required, such as after `?` in types
(php 8.0 union types forbid combining `?` with multiple types)

TODO: Check for any regressions in diagnostics

TODO: Validate that this can be used with real projects

Author: TysonAndre

README.md

@@ -2,13 +2,15 @@
 [![Build Status](https://travis-ci.org/Microsoft/tolerant-php-parser.svg?branch=master)](https://travis-ci.org/Microsoft/tolerant-php-parser)
 
 This is an early-stage PHP parser designed, from the beginning, for IDE usage scenarios (see [Design Goals](#design-goals) for more details). There is
-still a ton of work to be done, so at this point, this repo mostly serves as 
+still a ton of work to be done, so at this point, this repo mostly serves as
 an experiment and the start of a conversation.
 
 ![image](https://cloud.githubusercontent.com/assets/762848/19023070/4ab01c92-889a-11e6-9bb5-ec1a6816aba2.png)
 
+This is the v1 branch, which changes data structures to support syntax added after the initial release for php 7.0.
+
 ## Get Started
-After you've [configured your machine](docs/GettingStarted.md), you can use the parser to generate and work 
+After you've [configured your machine](docs/GettingStarted.md), you can use the parser to generate and work
 with the Abstract Syntax Tree (AST) via a friendly API.
 ```php
 <?php
@@ -38,17 +40,17 @@ foreach ($astNode->getDescendantNodes() as $descendant) {
         // All Nodes link back to their parents, so it's easy to navigate the tree.
         $grandParent = $descendant->getParent()->getParent();
         var_dump($grandParent->getNodeKindName());
-        
+
         // The AST is fully-representative, and round-trippable to the original source.
         // This enables consumers to build reliable formatting and refactoring tools.
         var_dump($grandParent->getLeadingCommentAndWhitespaceText());
     }
-    
+
     // In addition to retrieving all children or descendants of a Node,
     // Nodes expose properties specific to the Node type.
     if ($descendant instanceof Node\Expression\EchoExpression) {
         $echoKeywordStartPosition = $descendant->echoKeyword->getStartPosition();
-        // To cut down on memory consumption, positions are represented as a single integer 
+        // To cut down on memory consumption, positions are represented as a single integer
         // index into the document, but their line and character positions are easily retrieved.
         $lineCharacterPosition = PositionUtilities::getLineCharacterPositionFromPosition(
             $echoKeywordStartPosition,
@@ -59,15 +61,15 @@ foreach ($astNode->getDescendantNodes() as $descendant) {
 }
 ```
 
-> Note: [the API](docs/ApiDocumentation.md) is not yet finalized, so please file issues let us know what functionality you want exposed, 
+> Note: [the API](docs/ApiDocumentation.md) is not yet finalized, so please file issues let us know what functionality you want exposed,
 and we'll see what we can do! Also please file any bugs with unexpected behavior in the parse tree. We're still
 in our early stages, and any feedback you have is much appreciated :smiley:.
 
 ## Design Goals
 * Error tolerant design - in IDE scenarios, code is, by definition, incomplete. In the case that invalid code is entered, the
-parser should still be able to recover and produce a valid + complete tree, as well as relevant diagnostics. 
+parser should still be able to recover and produce a valid + complete tree, as well as relevant diagnostics.
 * Fast and lightweight (should be able to parse several MB of source code per second,
- to leave room for other features). 
+ to leave room for other features).
   * Memory-efficient data structures
   * Allow for incremental parsing in the future
 * Adheres to [PHP language spec](https://github.com/php/php-langspec),
@@ -83,34 +85,34 @@ so each language server operation should be < 50 ms to leave room for all the
  confusing, really fast, so readability and debug-ability is high priority.
 * Testable - the parser should produce provably valid parse trees. We achieve this by defining and continuously testing
  a set of invariants about the tree.
-* Friendly and descriptive API to make it easy for others to build on. 
+* Friendly and descriptive API to make it easy for others to build on.
 * Written in PHP - make it as easy as possible for the PHP community to consume and contribute.
 
 ## Current Status and Approach
 To ensure a sufficient level of correctness at every step of the way, the
 parser is being developed using the following incremental approach:
 
-* [x] **Phase 1:** Write lexer that does not support PHP grammar, but supports EOF 
+* [x] **Phase 1:** Write lexer that does not support PHP grammar, but supports EOF
 and Unknown tokens. Write tests for all invariants.
 * [x] **Phase 2:** Support PHP lexical grammar, lots of tests
-* [x] **Phase 3:** Write a parser that does not support PHP grammar, but produces tree of 
+* [x] **Phase 3:** Write a parser that does not support PHP grammar, but produces tree of
 Error Nodes. Write tests for all invariants.
 * [x] **Phase 4:** Support PHP syntactic grammar, lots of tests
 * [ ] **Phase 5 (in progress :running:):** Real-world validation and optimization
   * [ ] _**Correctness:**_ validate that there are no errors produced on sample codebases, benchmark against other parsers (investigate any instance of disagreement), fuzz-testing
   * [ ] _**Performance:**_ profile, benchmark against large PHP applications
-* [ ] **Phase 6:** Finalize API to make it as easy as possible for people to consume. 
+* [ ] **Phase 6:** Finalize API to make it as easy as possible for people to consume.
 
 ### Additional notes
 A few of the PHP grammatical constructs (namely yield-expression, and template strings)
 are not yet supported and there are also other miscellaneous bugs. However, because the parser is error-tolerant,
 these errors are handled gracefully, and the resulting tree is otherwise complete. To get a more holistic sense for
-where we are, you can run the "validation" test suite (see [Contributing Guidelines](Contributing.md) for more info 
+where we are, you can run the "validation" test suite (see [Contributing Guidelines](Contributing.md) for more info
 on running tests). Or simply, take a look at the current [validation test results](https://travis-ci.org/Microsoft/tolerant-php-parser).
 
-Even though we haven't yet begun the performance optimization stage, we have seen promising results so far, 
-and have plenty more room for improvement. See [How It Works](docs/HowItWorks.md) for details on our current 
-approach, and run the [Performance Tests](Contributing.md#running-performance-tests) on your 
+Even though we haven't yet begun the performance optimization stage, we have seen promising results so far,
+and have plenty more room for improvement. See [How It Works](docs/HowItWorks.md) for details on our current
+approach, and run the [Performance Tests](Contributing.md#running-performance-tests) on your
 own machine to see for yourself.
 
 ## Learn more
@@ -119,7 +121,7 @@ own machine to see for yourself.
 **:book: [Documentation](docs/GettingStarted.md#getting-started)** - learn how to reference the parser from your project, and how to perform
 operations on the AST to answer questions about your code.
 
-**:eyes: [Syntax Visualizer Tool](syntax-visualizer/client#php-parser-syntax-visualizer-tool)** - get a more tangible feel for the AST. Get creative - see if you can break it! 
+**:eyes: [Syntax Visualizer Tool](syntax-visualizer/client#php-parser-syntax-visualizer-tool)** - get a more tangible feel for the AST. Get creative - see if you can break it!
 
 **:chart_with_upwards_trend: [Current Status and Approach](#current-status-and-approach)** - how much of the grammar is supported? Performance? Memory? API stability?
 
@@ -131,10 +133,10 @@ operations on the AST to answer questions about your code.
   * [Validation Strategy](docs/HowItWorks.md#validation-strategy)
 
 **:sparkling_heart: [Contribute!](Contributing.md)** - learn how to get involved, check out some pointers to educational commits that'll
-help you ramp up on the codebase (even if you've never worked on a parser before), 
+help you ramp up on the codebase (even if you've never worked on a parser before),
 and recommended workflows that make it easier to iterate.
 
 ---
-This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). 
-For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact 
+This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
+For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact
 [[email protected]](mailto:[email protected]) with any additional questions or comments.

composer.json

@@ -3,10 +3,10 @@
     "description": "Tolerant PHP-to-AST parser designed for IDE usage scenarios",
     "type": "library",
     "require": {
-        "php": ">=7.0"
+        "php": ">=7.2"
     },
     "require-dev": {
-        "phpunit/phpunit": "^6.4|^7.5.20"
+        "phpunit/phpunit": "^7.5.20"
     },
     "license": "MIT",
     "authors": [

src/Node/CatchClause.php

@@ -7,23 +7,17 @@
 namespace Microsoft\PhpParser\Node;
 
 use Microsoft\PhpParser\Node;
+use Microsoft\PhpParser\Node\DelimitedList\QualifiedNameList;
+use Microsoft\PhpParser\MissingToken;
 use Microsoft\PhpParser\Token;
 
 class CatchClause extends Node {
     /** @var Token */
     public $catch;
     /** @var Token */
     public $openParen;
-    /** @var QualifiedName */
-    public $qualifiedName;
-    /**
-     * @var QualifiedName[]|Token[] Remaining tokens and qualified names in the catch clause
-     * (e.g. `catch (FirstException|SecondException $x)` would contain
-     *  the representation of `|SecondException`)
-     *
-     * TODO: In the next backwards incompatible release, replace qualifiedName with qualifiedNameList?
-     */
-    public $otherQualifiedNameList;
+    /** @var QualifiedNameList[]|MissingToken */
+    public $qualifiedNameList;
     /** @var Token|null */
     public $variableName;
     /** @var Token */
@@ -34,8 +28,7 @@ class CatchClause extends Node {
     const CHILD_NAMES = [
         'catch',
         'openParen',
-        'qualifiedName',
-        'otherQualifiedNameList',
+        'qualifiedNameList',
         'variableName',
         'closeParen',
         'compoundStatement'

src/Node/Expression/AnonymousFunctionCreationExpression.php

@@ -38,8 +38,7 @@ class AnonymousFunctionCreationExpression extends Expression implements Function
         // FunctionReturnType
         'colonToken',
         'questionToken',
-        'returnType',
-        'otherReturnTypes',
+        'returnTypeList',
 
         // FunctionBody
         'compoundStatementOrSemicolon'

src/Node/Expression/ArgumentExpression.php

@@ -10,15 +10,12 @@
 use Microsoft\PhpParser\Token;
 
 class ArgumentExpression extends Expression {
-    /** @var Token|null for php named arguments. If this is set, byRefToken and dotDotDotToken will not be set. */
+    /** @var Token|null for php named arguments. If this is set, dotDotDotToken will not be set. */
     public $name;
 
     /** @var Token|null */
     public $colonToken;
 
-    /** @var Token|null */
-    public $byRefToken; // TODO removed in newer versions of PHP. Also only accept variable, not expression if byRef
-
     /** @var Token|null */
     public $dotDotDotToken;
 
@@ -28,7 +25,6 @@ class ArgumentExpression extends Expression {
     const CHILD_NAMES = [
         'name',
         'colonToken',
-        'byRefToken',
         'dotDotDotToken',
         'expression'
     ];

src/Node/Expression/ArrowFunctionCreationExpression.php

@@ -40,8 +40,7 @@ class ArrowFunctionCreationExpression extends Expression implements FunctionLike
         // FunctionReturnType
         'colonToken',
         'questionToken',
-        'returnType',
-        'otherReturnTypes',
+        'returnTypeList',
 
         // body
         'arrowToken',

src/Node/FunctionReturnType.php

@@ -11,10 +11,9 @@
 trait FunctionReturnType {
     /** @var Token */
     public $colonToken;
+    // TODO: This may be the wrong choice if ?type can ever be mixed with other types in union types
     /** @var Token|null */
     public $questionToken;
-    /** @var Token|QualifiedName */
-    public $returnType;
-    /** @var DelimitedList\QualifiedNameList|null TODO: Merge with returnType in a future backwards incompatible release */
-    public $otherReturnTypes;
+    /** @var DelimitedList\QualifiedNameList|null */
+    public $returnTypeList;
 }

src/Node/MethodDeclaration.php

@@ -34,8 +34,7 @@ class MethodDeclaration extends Node implements FunctionLike, ModifiedTypeInterf
         // FunctionReturnType
         'colonToken',
         'questionToken',
-        'returnType',
-        'otherReturnTypes',
+        'returnTypeList',
 
         // FunctionBody
         'compoundStatementOrSemicolon'

src/Node/Parameter.php

@@ -6,6 +6,7 @@
 
 namespace Microsoft\PhpParser\Node;
 
+use Microsoft\PhpParser\MissingToken;
 use Microsoft\PhpParser\Node;
 use Microsoft\PhpParser\Token;
 
@@ -16,13 +17,8 @@ class Parameter extends Node {
     public $visibilityToken;
     /** @var Token|null */
     public $questionToken;
-    /** @var QualifiedName|Token|null */
-    public $typeDeclaration;
-    /**
-     * @var DelimitedList\QualifiedNameList a list of other types, to support php 8 union types while remaining backwards compatible.
-     * TODO: Merge with typeDeclaration in a future backwards incompatible release.
-     */
-    public $otherTypeDeclarations;
+    /** @var DelimitedList\QualifiedNameList|MissingToken|null */
+    public $typeDeclarationList;
     /** @var Token|null */
     public $byRefToken;
     /** @var Token|null */
@@ -38,8 +34,7 @@ class Parameter extends Node {
         'attributes',
         'visibilityToken',
         'questionToken',
-        'typeDeclaration',
-        'otherTypeDeclarations',
+        'typeDeclarationList',
         'byRefToken',
         'dotDotDotToken',
         'variableName',

src/Node/PropertyDeclaration.php

@@ -6,9 +6,11 @@
 
 namespace Microsoft\PhpParser\Node;
 
+use Microsoft\PhpParser\MissingToken;
 use Microsoft\PhpParser\ModifiedTypeInterface;
 use Microsoft\PhpParser\ModifiedTypeTrait;
 use Microsoft\PhpParser\Node;
+use Microsoft\PhpParser\Node\DelimitedList\QualifiedNameList;
 use Microsoft\PhpParser\Token;
 
 class PropertyDeclaration extends Node implements ModifiedTypeInterface {
@@ -20,14 +22,8 @@ class PropertyDeclaration extends Node implements ModifiedTypeInterface {
     /** @var Token|null question token for PHP 7.4 type declaration */
     public $questionToken;
 
-    /** @var QualifiedName|Token|null */
-    public $typeDeclaration;
-
-    /**
-     * @var DelimitedList\QualifiedNameList|null
-     * TODO: Unify with typeDeclaration in a future backwards incompatible release
-     */
-    public $otherTypeDeclarations;
+    /** @var QualifiedNameList|MissingToken|null */
+    public $typeDeclarationList;
 
     /** @var DelimitedList\ExpressionList */
     public $propertyElements;
@@ -39,8 +35,7 @@ class PropertyDeclaration extends Node implements ModifiedTypeInterface {
         'attributes',
         'modifiers',
         'questionToken',
-        'typeDeclaration',
-        'otherTypeDeclarations',
+        'typeDeclarationList',
         'propertyElements',
         'semicolon'
     ];