Development UpdatesUpdate 2015-08-18
With some DCPU-16-containing projects showing significant progress and momentum (most notably, Tech Compliant), I expect to begin ramping up work on DevCPU. If I recall correctly, the repository is still in an unstable state. There's a lot of work still to be done, though, if the goal were simply to stabilize the current feature set, a version could probably be packaged up fairly quickly. As it is, DevCPU is built on a pre-release version of Eclipse Kepler. I had been putting off updating it to Kepler SR1, but I think it's best to bite the proverbial bullet and, potentially, skip over Luna and bring it all the way up to Eclipse Mars, which released just a few weeks ago. This will likely break quite a lot, but only in terms of the rich client platform. The emulation, assembly, and other logical components will be unaffected; it's just the components that tie together the interface that will need to be reworked, and some of them are overdue for a rewrite anyway. We'll see. My current plan is to, sometime very soon (probably this week), get Mars and pull the repository, evaluate the state of functionality of the codebase in that environment and determine whether to proceed from there or to fall back to an earlier version of Eclipse and continue development as I have previously.
I've been working on extension points so that people can start making their own hardware, preprocessor, assembler, etc. for DevCPU. The devcpu.hardware extension point is partially implemented and is being used by the Device Manager. I actually never upgraded to SR2. I may still, but I experimentally upgraded to Luna RC2 and too much was...off, so I've abandoned the idea of upgrading the eclipse environment for a while. Work is a little slow, as I've been sucked into working on a DCPU mod for Minecraft. On that front, the first major technical hurdle (getting LEM1802s to render in-world) has been passed, and the next hurdle will be getting hardware devices connected to DCPUs (which have already been tested to run in-world using tile entity ticks). You can follow progress on "DCPUCraft" at its github page.
I'm updating my Kepler from the prerelease version I started DevCPU with, to SR2. I'm going to start looking at creating extension points for stuff like hardware/preprocessor/etc. so I can possibly include some of that extensibility in the first "real" release. I'm also hoping the update might help me with the problems I was having getting proper breakpoints to work in the platform and DCPU code editor..
I've been so lazy lately, but I'm about to get back to it. First, I'll be finishing the hex view and then getting breakpoints working.
The preprocessor extraction has been completed successfully. Initial indications suggest that I was successful in accomplishing this without having a negative impact on assembly speed. At first attempt, dcpu-admiral assembled properly in 1704 ms with all 1328 possible short literal values optimized, which is consistent with its assembly speed prior to the changes. There are now different phases of assembly than there were at Release 0. Here they are, listed with split times for assembly of dcpu-admiral.
1. Line loading / Preprocessing (1098ms)
2. Tokenizing / Analyzing (390ms)
3. Label gathering (15ms)
4. Sizing / Locating (38ms)
5. Assembling (159ms)
I'm quite pleased with the performance, as I have been for quite some time. Reading the files from the disk continues to take up half of the assembly time.
The change from the all-in-wonder magic assembler to a more graduated assembly process with discrete steps is driven by two long-term goals. Firstly, it will allow new features like multi-line macros to be more easily implemented, and secondly, it constitutes a step toward a more extensible IDE. The intent, with DevCPU, is to get to the point where there are extension points a-plenty, including extension points for preprocessors and even assemblers. The new preprocessor implements a simple interface and people wishing to use their own preprocessor will eventually be able to write a class that implements it and write a simple plugin to make their preprocessor available in DevCPU, using RCP extension points. Many things, from the assembler, to the hardware, to the views, and so on, will be pluggable in the same manner, so that, for example, you can write your own hardware for DevCPU and share it as a plugin that anyone else can add to their installation.
I've been working on the preprocessor. Updating this development log hasn't been much of a priority.
Go away; I'm being lazy.
Tonight, now that the debugger will switch between source files, I'm going to try to get the correct file offsets to the debugging environment so the instruction pointer won't get off track anymore. Once that's done, I think I'll start on some changes to the assembler. I'm going to pull out an independent preprocessor. The all-in-wonder magical assembler was nice, but for the sake of making some of the upcoming changes (like macros and conditional compilation directives) simpler, I'm going to head the more traditional route of having a separate preprocessor. Later on in DevCPU's development, I hope to make it possible to plug in other preprocessors via extension points. This change, pulling the preprocessor out, shouldn't actually have a noticable impact on assembly speed if I do it right. Or at least, that's what I'm hoping.
Doing too much awesome work on DevCPU to remember to update this every day. Will be more detailed tomorrow.
After spending most of the week not doing much, I've started to finish the debugging environment that I left incomplete at Release 0, and things are progressing quite nicely. I've been able to hook up the DCPUs so that I can control them using RCP's native debugging controls (suspend/resume/step) and I also have it showing a marker at the line of code that corresponds to PC's current value.
When I've fleshed out that functionality a little more, I'll be done with the core, critical debugging features and I'll start working on some of the next tier of debugging featurese like breakpoints and live memory view, both of which have some parts already implemented. After that, I'll work on the third tier of debugging features like watches, value popups on hover in the code, and those sorts of things. Things are looking good and I expect to be able to get these features done faster than I originally thought.
You can expect me to resume updating this development log daily, starting today.
I'm considering publishing an interstitial release after completion of the core debugging features so that people can start using them sooner. If I do, it will just be labeled Release 0 Update 0, like the current preview release, and will replace it on the download page.
...just to screw with people.
I've been slowly building from Release 0 Update 0 and I'm about to kick Release 1 development into gear. I'll first be focusing my efforts on implementing the source lookup for the debug environment so I can start getting debugging features in place. Another task on the near-term list is reworking the assembler yet again. With this rework, I intend to pull some tasks out into a separate preprocessor. One reason for this is to make it possible, in the future, to swap in a different preprocessor and to make configuring any preprocessing options easier. The current assembler does a unique little all-things togetherish thing and, although it is slightly faster that way, it no longer seems necessary. DevCPU is already far faster at assembling than the other popular alternatives, so trading off to get a more configurable preprocessor for a very slight speed hit sounds good to me.
I've published new builds on the Download page. This is primarily an update to address problems some people were having, getting the Mac version to load, but it also updates LWJGL to the last successful nightly build and removes the interpolation from the LEM1802 that made it look fuzzy when it was displayed large.
I've published Release 0 Update 0 again. I've cut platforms down to Windows, Mac, and Linux. I've also dropped Motif on Linux and WPF on Windows. This had less to do with compatibility and more to do with not having so many unnecessary builds. There are now only 9 platform versions plus the Eclipse plugin version. Head over to the Downloads page to get Release 0 Update 0. I had intended to remove the interpolation from the LEM1802 while I was preparing the re-release, but I didn't get around to it, sorry. Release 1 will have a more sensible approach to scaling the display, to keep it from being fuzzy when it's large and from being jaggy when it's small.
NOTICE: If you have downloaded Release 0 Update 0, you got a bad version. You do not have Release 0 Update 0. Remove it. Once this build issue is resolved, there will be an announcement of the re-launch of Update 0.
Release 0 Update 0 is now available. Head over to the Download page to get it. I have dropped support for Mac OS X Carbon with this update. It is now Cocoa only for Mac.
Oh right, updates. I've been working through the last few reported issues. Lucus has been helping me test on Linux. The major holdup has been a problem that popped up recently that causes the application to hang under unknown, but not difficult to cause, circumstances. I'm somewhat hopeful about my most recent attempt to correct this, though it seems that Linux is still having problems. It's hard to tell when this freezing problem is fixed since it's difficult to test. You can only really know when it isn't fixed. I'll have it resolved soon, one way or another, and I'll put out Release 0 Update 0, so I can get to work on Release 1.
After seeing the trouble people on some platforms were having, I decided to allow a small window of support and accept issues (on GitHub Issues) to be addressed in an upcoming (quite soon) Release 0 Update 0. Most issues are already fixed for the update, but if you don't see yours in Issues, make sure you submit the report immediately, as support is about to end, and will not continue again until after Release 1. I stopped updating this page for a few days, while I was working on some of those issues, but I'll start updating it again, dailyish, as I was before.
No reason to wait, when everything's ready for download. Just a reminder, DevCPU isn't done yet. This is a thrown-together rush pre-release preview version, called "Release 0".
I've been updating config files, adding missing icons, and other such pre-release busy work. Tomorrow is final cleanup and packaging prep.
I've completed the third and final phase of the multipass Assembly switch. Full short literal optimization is now built-in to the assembly process due to iterative sizing passes. The new assembly strategy assembles in approximately the same amount of time as the old assembler (faster in some instances, the same or slightly slower in others) but does full short literal optimization. The latest version of dcpu-admiral (13430 lines in 18 files) as retrieved from its repository a few minutes ago, loads and assembles to a DCPU in DevCPU in a total time of 1.53 seconds on this laptop, producing a binary identical in size to that which is produced by Organic with its short literal optimization turned on. Although I've never used Organic, orlof reports that, in his own tests, Organic takes 1,030 seconds to assemble dcpu-admiral with S.L.O. turned on. If orlof's test system is comparable to this laptop, then DevCPU's new assembler is apparently able to assemble large projects, with full S.L.O., over 500 times faster than Organic. Of course, DevCPU can't currently assemble from a command line, so I'm not sure such a direct comparison is appropriate between those assemblers.
Anyway, the next couple of days will be mostly cleanup, tying up loose ends, packaging, and other preparations for Release 0 on 1 April 2013.
I'm in the process of a major overhaul of the Assembly process, moving to a multipass, optimized strategy. I'm nearly done but there is still some work to do. It looks like the assembly speed will remain about the same, but account for any possible short literal optimizations.
Preparing for Release 0.
I've committed an initial memory/disk hex view solution, though I haven't really wired it in to everything yet. I'm going to hold off on doing that until the end of this sprint, because it isn't a priority. The long term plan is to get the RCP's MemoryView working properly. I nearly had it but hit a wall and so I'd decided to put it off until after Release 0. The plan is for this temporary hex memory/disk view solution to be available in Release 0. I'm going to work out a schedule for the remaining days in this sprint so I can be sure to leave enough time to wrap things up, do some housekeeping, and work out the packaging of the different versions of DevCPU. I might start a minor, non-destructive redesign of the assembly passes right now, in preparation for some of the optimizations already planned.
I did a lot of heavy restructuring in the assembler and made it run in a Job with a dialog. It's a lot of really detailed and specific things, so I'm not going to go into the details here. I didn't get to working on source lookup yet, so I guess I should do that next. Instead, however, I'm probably going to do some more work on the assembler, maybe make it do more short literal optimizations, before getting to source lookup. Still on track for a 1 April 2013 Release 0.
I've been working heavily on launch/debug integration. I have launches being created and I'm tied into the debug system now, but I still can't get the Memory View to work. The Registers view works though (not that you'd ever use it; the simple view I wrote myself is more useful than that). I'm going to stop worrying about it for now as it's consuming too much development time. I'm considering writing my own memory view. While not as configurable as the one in RCP's debugging suite, I can at least get something usable in place. I suppose I should get the source lookup fleshed out so I can turn on breakpoints and stepping and everything. I'm not entirely certain how that should work. I guess I'll use the mappings generated by the assembler to find the code in the right file. It will get weird when things like macros get supported, and lines aren't really single instructions anymore. Whatever works.
I've added support for Defines being used inside directives (including other defines). After that, I was able to checkout, assemble from source, and use the latest DCPU-Admiral beta, all from within DevCPU. orlof's comments in the past lead me to believe that that means DevCPU is awesome. I'm not thrilled about how I had to add that support, so it will probably warrant revisiting some time after Release 0. I'm tempted to work on adding macros, but I really think I need to get launch/debug integration going, so I'm going to turn my focus to that.
The new assembler is complete. DATs are now assembled. I've also added support for .fill and fixed some bugs. I did a lot of testing, and though I'm not yet satisfied and want to do more testing to make sure everything is working properly, I believe I'll go ahead and move on to integrating with the RCP's launch/debug system, and get started on that so I can come up with a target date for Release 0.
The new assembler is complete, except for DATs. It also hasn't been tested. I'll be adding in the DAT data and then testing it, first with a variety of programs, and then with some tests of specific functionality.
I'm working on expression parsing. It's going well, though I expect I'll be revisiting this after Release 0.
I've moved on from the preprocessor. I added the missing unary operator tokenizaion in the lexer. I've started work on the assembly pass of the assembler. Currently, I'm hashing out the evaluation of the instructions' values/expressions. I have a pretty concise plan for how to do this last part of the new assembler and am in the process of implementing it. The assembler rewrite should be finished very soon. From there, it's integrating with the RCP's launch/debug system, then a few housekeeping items, and I'll be ready for release 0. I will continue to stream much of its development on twitch.tv/HerobrinesArmy, since some of you, apparently, get very, very bored sometimes.
A lot of progress on the preprocessor. Local .labels are now supported, as is .origin, .align, .reserve, .define, and .include. I'm to the point where every line knows both how large it will be (if it will even occupy memory) and at what offset.
The new assembler is progressing rapidly. The include system is in place and is extremely flexible in terms of how you can specify the location of a file to include (both in the way you define the path itself and in how you write the line), though paths are case sensitive. The include system also checks for non-existent files and for recursive inclusions (by checking the current file's ancestry for the file to be included). Label definitions and uses are collected and tracked. All lines are read, tokenized, and mapped in and associated with the document that contained them.
So, most of the hard stuff is done except for handling the (already tokenized) expressions, and handling whatever other directives I'm going to add. The actual assembly of the code will mostly rely on leveraged code from the old assembler.
The lexer is complete for now. Post-release, additional operators will be tokenized as support for them is added, but for now, it's complete. A tentative assembly strategy has been selected and work has begun to implement it. The assembler rewrite has begun.
The lexer is mostly complete. It needs some additional tokens but it's good enough for now to say that it's no longer something to be tackled. I'm transitioning to a new focus, which is evaluating different approaches to creating the new assembler. I need it to meet a lot of constraints, not all of which are immediately relevant, but all of which will eventually come into play. The biggest roadblocks I see are choosing an optimal strategy for includes, taking into account all of those constratints, and an optimal approach for handling expressions in the assembly process.
I've developed the new lexer to a point where it's somewhat usable. It's handling labels, instructions, and comments at the moment. Literals can be decimal (-12356), hexadecimal (0xd25F), binary (0b1110010110011101 ), or even character ('#'). Grouping and expressions are parsed (4 operators for now; many more to be added later). Most whitespace is optional, as are commas separating values. Labels work whether the colon is prefixed or postfixed. I still need to add in a few pieces like PUSH et al, DAT lines, and directives, but the approach I've taken toward building the lexer appears to be proving quite successful and it seems to process quite quickly. Here's an example of how the lexer is working at the moment. The lexer can handle the following line:
From that line, it produces the following structure:
:label1 label2: set[some_label+0x10-'~' ](( 11+2)-(a)+(2/(0b1010101010101010)));commentssss
LabelDefinitionToken (:label1) LabelDefinitionToken (label2:) BasicOpCodeToken (set) BValueStartToken () AddressStartToken ([) LabelToken (some_label) OperatorToken (+) LiteralToken (0x10) OperatorToken (-) LiteralToken ('~') AddressEndToken (]) BValueEndToken () AValueStartToken () GroupStartToken (() GroupStartToken (() LiteralToken (11) OperatorToken (+) LiteralToken (2) GroupEndToken ()) OperatorToken (-) GroupStartToken (() RegisterToken (a) GroupEndToken ()) OperatorToken (+) GroupStartToken (() LiteralToken (2) OperatorToken (/) GroupStartToken (() LiteralToken (0b1010101010101010) GroupEndToken ()) GroupEndToken ()) GroupEndToken ()) AValueEndToken () CommentToken (;commentssss) EndOfLineToken ()
Next, I'll be adding some missing bits to the lexer, then I'll spend a little time cleaning up the syntax coloring, getting everything using the preference store so it can all be changed by the user (I'll hold off on fleshing out the actual preference dialog page until later), then I'll be integrating the DASM files into the RCP's "build" system (this is when the assembler rewrite will begin in earnest, leveraging the power of the lexer). Then I'll integrate it with the RCP's "launch" system, adding debugging features at that time. After that, I'll look over things, get any missing functionality in the code editor handled, do a little cleanup, and publish the first release of DevCPU.
It may sound like a lot, but I believe it's best to accomplish those things before release, and I don't expect it to take very long. The beauty of building an IDE on Eclipse's Rich Client Platform is that most of the hard/cool stuff is already done for you. Most of the work is done in integration, things like getting the DCPU emulator to work with RCP's launch/debug system so all of RCP's debugging features (breakpoints, hover values, memory inspection/modfication, watches, stepping, etc.) will suddenly be availabe to the user in the same manner that it is available to them when doing Java development in Eclipse. There's a reason Eclipse is the IDE of choice for Java developers, and it's the powerful, flexible, convenient feature set that makes both coding and debugging easier and faster. That's what DevCPU is intended to deliver.
I've abandoned Xtext entirely, meaning I'm having to write the Code Editor, lexer, parser, partitioner, damager/repairer, etc. myself. I had a slow start on that but now things are progressing quite nicely. I already have a usable editor with some syntax coloring, but I still have a good bit of work left before the editor will be "complete" and have all of the nice features it had before.
Once that's done, I hope to leverage the lexer/tokenizer/partitioner, written for the code editor, as a foundational piece of the new assembler, and then to feed the debugging suite with source code location information when I move into writing the launch system integration so it will have all of those nice interactive debugging features you might want.