SharpDevelop Community

Get your problems solved!
Welcome to SharpDevelop Community Sign in | Join | Help
in Search

How does #D IntelliSense/Refactoring compare to Roslyn?

Last post 07-21-2012 5:05 PM by DanielGrunwald. 7 replies.
Page 1 of 1 (8 items)
Sort Posts: Previous Next
  • 07-17-2012 9:47 PM

    How does #D IntelliSense/Refactoring compare to Roslyn?

    I don't know anything about #D's architecture, but I'm curious how it compares to Roslyn. For instance

    • what are the names of the parts (libraries/components) that support intellisense & refactoring in #D?
    • is there a common architecture for intellisense & refactoring? Could it be used for compiling too?
    • is there a common architecture used by multiple programming languages?
    • are the syntax trees immutable and do they resemble Roslyn's red-green trees?
    • which architecture do you think is better? Is #D learning anything from Roslyn?

     

  • 07-20-2012 10:09 PM In reply to

    Re: How does #D IntelliSense/Refactoring compare to Roslyn?

    In SharpDevelop 4.x, the NRefactory library contains the C# parser, and the 'SharpDevelop.Dom' library contains the type system, semantic analysis and code completion (intellisense). Refactorings are spread across various SharpDevelop addins.

    In 2009, we started rewriting all of that code, making some massive architecture changes - this rewrite is NRefactory 5 (which now includes the type system etc.). It turned out to be surprisingly similar to Roslyn. NRefactory 5 is stable and already used by MonoDevelop 3.0. However, we haven't finished porting SharpDevelop yet - we plan to do so for SharpDevelop 5.0 ('newNR' branch on github).

    Intellisense and refactoring both are included in NRefactory 5 and build upon the C# parser and semantic analysis engine. Yes, the semantic analysis provides enough information for compiling. SaltarelleCompiler is a C#-to-JavaScript compiler using NRefactory 5 as a front-end.

    The NRefactory type system is language-independent, and several ResolveResults (representation of program semantics) are also language-independent. But most of the code is C#-specific: the syntax tree, semantic analysis logic, refactoring engine etc.

    NRefactory C# syntax trees are mutable (but can be frozen to prevent accidental mutations).

    The NRefactory architecture already was pretty complete before the first Roslyn CTP was published, so we couldn't learn much there. But the architectures turned out to be pretty similar, I don't see Roslyn having any major advantages.

    I'm not really decided on which syntax tree concept is better. In ILSpy where we make extensive use of pattern matching and AST transformation, I prefer having a mutable AST. For SharpDevelop, immutability would be nice. But Roslyn is lacking AST pattern matching, and constructing the syntax tree is a bit more verbose (with NRefactory, you don't need to explicitly specify the tokens). Refactorings seem to be easier to implement using NRefactory.

  • 07-21-2012 12:15 AM In reply to

    Re: How does #D IntelliSense/Refactoring compare to Roslyn?

    Thanks for the detailed answer!

    I like persistent trees the most, at least in principle, although I am also concerned about performance. So it's hard to say what's best without some benchmarks. A nice compromise between the two is "fast cloning". I don't know if such a design could fit into NRefactory (without looking at it more closely), but for instance I developed a B-tree like data structure called AList<T> which supports O(1) cloning and freezing (and when you clone a frozen tree, you get an unfrozen tree out of it.) Then it does copy-on-write for parts of the tree that are changed later. I will try to put up a CodeProject article about it soon. AList<T> itself is probably not useful inside NRefactory (e.g. it's designed for large, linear lists) but I think the general concept of being able to mutate parts of a tree and then create clones instantly is a good compromise for thread-safety needs.

    After looking at Roslyn briefly I got the feeling they like things to be a little bit too verbose, I mean: ClassDeclarationSyntax instead of just ClassDecl or ClassDeclaration? CommonSyntaxToken instead of just, er, Token? Other than that--and the lack of common code between languages (MS always prefers to throw more manpower at a problem instead)--Roslyn seems like a pretty solid design and any convergence with NRefactory seems like a good thing (so that "porting" a tool between NRefactory and Roslyn is not excessively difficult.)

    I'm considering creating an enhanced version of C# and/or a .NET version of D--actually my concept is more ambitious than that--but it looks like Roslyn is closed for creating new or enhanced languages; for instance the constructor of CommonSyntaxToken is internal (with InternalsVisibleTo => MS C# and VB compilers). So probably I will look for an approach starting with NRefactory.

    Tell me, are you interested in giving NRefactory a kind of 'universal type system' and other features so it can support arbitrary languages? And how far is it from being able to at least support VB (whose type system is very similar to C#)? Not that I really care about VB, I'm just asking how well the architecture can cope with multiple languages. And one of the things I want to see happen is multiple languages interoperating seamlessly: imagine if you could drop C# and C/D/VB/F#/Javascript files in a single project, compile them to a single DLL and easily call between languages, sharing data structures and all that. So it's like the CTS implemented at the compiler level, but more powerful (since CTS is missing plenty of elements that multiple languages have, such as slices and algebraic types.)

    That's half of my vision; the more important half is an extensible language with powerful compile-time metaprogramming, and support for code-completion for such a language requires that the IDE run code at compile-time, in the background, automatically, because some of the classes and methods of the program would be generated based on the results of other code that runs at compile-time. Thus, the IDE must essentially be a compiler itself. I'm not actually interested in making a new programming language, I just want to make the languages we've already got more productive.

    I may be interested in helping to develop such features for NRefactory, if you are receptive.

    The pattern-matching looks promising; I had been planning to create some kind of parser generator with tree parsing similar to what ANTLR has, but the NR's pattern-matching system would certainly reduce the need for such a thing.

  • 07-21-2012 11:24 AM In reply to

    Re: How does #D IntelliSense/Refactoring compare to Roslyn?

    A mutable AST also simplifies the API - we don't need to remember the root node for performing changes; and we can use object initializer syntax for creating new nodes. Performance isn't my main concern here, even in ILSpy most of the time is spent analyzing the code, actually performing AST changes is only a small part.

    If I had to write a new parser and AST from scratch, I'd probably use Roslyn-style immutable trees. This would also allow building an incremental parser (I've already experimented with this in NRefactory.Xml).

    I don't really get the point of the CommonSyntaxToken - what do you gain from having a common base class for the syntax trees of multiple languages? I think any operations on the syntax tree are going to be language-specific anyways.

    Type system: NRefactory allows custom IType implementations. And I wouldn't be opposed to adding commonly used types that the CTS is missing to the NRefactory core. We already have anonymous types and Java-like intersection types (Interface1 & Interface2) in there.

  • 07-21-2012 2:09 PM In reply to

    Re: How does #D IntelliSense/Refactoring compare to Roslyn?

    DanielGrunwald:
    I don't really get the point of the CommonSyntaxToken - what do you gain from having a common base class for the syntax trees of multiple languages? I think any operations on the syntax tree are going to be language-specific anyways.

    It's a struct actually - two references and two ints to define the token - which is convertible to (and from) a language-specific token struct.

    The Roslyn people said the same thing in justifying why they had two completely independent syntax trees for C# and VB. I don't understand this POV at all; both languages have if-statements, while, foreach, similar operators, variable declarations, methods, properties, classes, structures, enums, delegates, and aren't things like overload resolution rules pretty similar? Sure, there are things that work differently, but there is surely enough in common for some refactorings and I would think, after lowering certain constructs to simpler ones, and after eliminating any special datatypes, most languages could share a common AST=>CIL backend. Am I missing something?

  • 07-21-2012 2:49 PM In reply to

    Re: How does #D IntelliSense/Refactoring compare to Roslyn?

    Qwertie:
    both languages have if-statements, while, foreach, similar operators, variable declarations, methods, properties, classes, structures, enums, delegates, and aren't things like overload resolution rules pretty similar?

    Syntactically, they are pretty different once you look into the details. For example, C# supports declaring multiple events on one line "public event EventHandler Click, DoubleClick;", VB doesn't. We tried using a common AST for C# and VB in NRefactory 4, but it was a mess, pretty much all code consuming the AST had to use "if (visualBasic)" statements to special-case stuff.

    And that was a simple abstract AST without token information. Once you add those, it becomes impossible to share syntax constructs between languages without making the AST API a horrible mess.

    Overload resolution: there are some similarities, but plenty of differences as well. IMHO two cleanly separated code bases are easier to maintain than a mixture of multiple languages.

    There are massive semantic differences between C# and VB once you get into the details. E.g. the equality operators work differently in many aspects; the division is different, member lookup rules are different, etc... The shared implementation we used up to SharpDevelop 4 just didn't work out, so we decided to split it up for NRefactory 5.

  • 07-21-2012 4:48 PM In reply to

    Re: How does #D IntelliSense/Refactoring compare to Roslyn?

    Hmmm...... I still am not really seeing it, because it seems like a few special cases is better than having to rewrite very similar code twice. Are there some code files you could point me to that smell especially bad with lots of special cases?

    The puzzle to me is, while clearly these differences are important to some clients (obviously, some code must account for differences in equality and division, member lookup rules, and how many events you can declare in one statement), there are other clients that don't care. I am imagining some kind of lookup service that resolves the meaning of any operator or method call in a language-specific manner, that would allow many clients not to care about differences in operators or overloading rules. The lookup service could have two separate implementations for C# and VB, but that's not important to clients. I wonder if even quite different constructs like 'foreach' and 'for', or between 'if' and 'switch' could be abstracted away at some level into just 'loop' and 'conditional'...

  • 07-21-2012 5:05 PM In reply to

    Re: How does #D IntelliSense/Refactoring compare to Roslyn?

    Most of the C# semantic analysis logic is actually independent from the AST (see CSharpResolver class). Other language implementations could call into that to re-use pieces that are identically.

    The main problem with a common AST is that I don't see any way to have a clean API for that. For example, look at the syntax of query expressions in C# and VB, and try to come up with a common syntax tree for both of them. And there's really tons of language constructs with subtle differences like this.

    It's easy to claim that a C# foreach and a VB For Each are similar enough, but that only really applies to clients that don't care - others will have trouble with the subtle differences (e.g. VB can use a field as loop variable). But every client will care about some AST node types. An AST that closely represents the input language can share only very few nodes (I'd guess less than 10%), an abstracted AST won't have enough details for those few node types that the client cares about.

    Edit:

    Qwertie:
    I would think, after lowering certain constructs to simpler ones, and after eliminating any special datatypes, most languages could share a common AST=>CIL backend

    I probably should mention that sharing a backend actually is possible. For example, SharpDevelop 5 has a single NR5-based interpreter that allows expression evaluation in the debugger - for C# and for future other NR5 languages. This works because the interpreter doesn't actually use the AST - it instead traverses the semantic tree (ResolveResults), which is language-independent.

Page 1 of 1 (8 items)
Powered by Community Server (Commercial Edition), by Telligent Systems
Don't contact us via this (fleischfalle@alphasierrapapa.com) email address.