Vulcan and Firebird merging plan

Dmitry Yemanov posted to the Firebird-devel list his proposals for merging the Firebird 2 code with the Vulcan code.

Here it comes, with comments from others:


Key Points
----------

1) The result of the merge is a new CVS tree
2) Modules refactored by Jim are taken from the Vulcan tree
3) The rest is taken from the FB tree
4) All the pieces are adjusted to fit each other
5) Vulcan's code is fixed to use the project coding rules


Details
-------

1) The common library will mostly based on the FB
class library with some classes taken from Vulcan (QSort, SyncObject,
StatusVector, etc). The common string class is Firebird::String. The
exception handling is based on the FB classes with some changes. Based
on Alex's research, I conclude that the memory manager should be taken
from Vulcan and adapted to the FB pool infrastructure. The configuration
stuff is also taken from Vulcan.

Responsible:
- Dmitry (syncs and other platform specifics)
- Alex (memory management, classes library, exceptions)
- Adriano (config stuff) -- ???

2) The Y-valve is taken from Vulcan and fixed to fit the new common
library and the project coding conventions. An exception handling and
MT-safety review is required, as well as some cleanup.

Responsible: Dmitry

3) The lock manager is taken from Vulcan, cleaned up and reviewed. I
don't agree with the way how the two cache modes are implemented there,
it looks ugly. Also, we need to carefully examine all the Windows and
Solaris specific changes made there.

Responsible: Dmitry

4) The JRD files are merged from the both trees. The page cache code and
the metadata classes (as well as some other refactored code like blob
classes, the mover, etc) are taken from Vulcan with a careful review.
Other files (cmp/exe/dfw/etc) are taken from FB and adjusted to the
Vulcan synchronization and metadata cache classes.

Arno Brinkman commented: opt, btr, idx, Rsbxxx modules should be inherited from Vulcan IMO.
Only missing pieces are the INTL stuff, which is not that big.
//end of comment

Dmitry Yemanov commented: As for Rsb modules, I agree. The same for opt.cpp and Optimizer.cpp, as
they're already in sync with HEAD. I just want them to be ported at the end.

But why do you want to preserve Vulcan's btr.cpp and idx.cpp? AFAIK,
HEAD is more up-to-date in those parts (and more patches are in the way).
//end of comment



Responsible:
- Dmitry (synchronization, metadata, other)
- Vlad (page cache, metadata)
- Alex (execute statement, authentication)
- Adriano (INTL stuff, other)

5) The DSQL files are taken from Vulcan, with a total review of the
metadata cache integration. This is the hardest part of the merge, IMO.
I'm not satisfied with the half-way DSQL integration done in Vulcan, but
I'm not sure we can offer something better in the short term. Changes in
pass1/gen/make are ported from the FB codebase. The parser generator is
most likely to be migrated from FB, it just lacks the MT-safe skeleton
implementation. A replacement with a hand-made RDP can be considered in
the long term.

Arno Brinkman commented: Although it's for the long term, i send you something for review a long time ago?
//end of comment

Dmitry Yemanov commented: Yep, and I didn't have time to comment on it properly. Mea culpa.
//end of comment

Responsible: Vlad, Dmitry, Adriano, Alex

6) The remote subsystem is taken from Vulcan, unless Alex objects. The
multi-client mode related fixes by Alex, protocol extensions by
Alex/Vlad/myself and various bugfixes should be backported from FB.
Amount of our changes there is not very big, so it looks doable.

Responsible: Alex, Vlad, Dmitry

7) The entire INTL aware code is taken from the FB codebase, a proper
MT-related review is to be performed.

Responsible: Adriano

8) Utilities code is to be taken from FB, the only merge-related thing
is an interface with the Vulcan services provider. The UDF libraries are
taken from FB. Guardian is to be considered for dying.

Responsible: ?


Special Notes
-------------

1) The non-shared-cache mode is preserved, but it should be
implemented using the config switch rather than #ifdefs. In particular,
it means that synchronization primitives should unconditionally exist in
all engine favors.

2) The internal DSQL implementation (FbDbc) is not ported from Vulcan.
We should either make it much more light-weight (to not carry the
redundant semantics like metadata representation), or consider some
other connectivity code, or keep GDML in the sources for the time being.
Any volunteer for this task?

3) I don't expect any serious INTL issues, as most of the INTL aware
code will be taken from FB, but I expect Adriano to track all the
related changes.

4) The RSB refactoring is ported after all else is done. Being not
critical part of Vulcan/FB3, it was one of the most buggiest parts of
Vulcan and I'm not sure we solved all the issues. Also, Arno intends to
make some improvements based on that code in the meantime.

Arno Brinkman commented: Almost done, only more testing and some refactoring left.
//end of comment

Dmitry Yemanov commented: That's great :-)
//end of comment


5) If anybody not explicitly counted as an assignee above would like to
assist in some parts, it would be appreciated.

6) New code is committed into CVS on a per subsystem basis.


Merge Steps
-----------

1) HEAD is cleaned up to follow the new threading rules. It mostly means
getting rid of the non-MT code branches (including signal-based ASTs).
We could also perform whatever global cleanup we consider useful. It
could be done in a separate CVS branch, if we'd like to keep the cleaned
version reserved for production.

Status: 80% done (in my private tree).

2) Lock manager is replaced with the Vulcan's one, we perform some dbb
adjustments and global synchronizations. Scheduler is removed. At this
point we have a skeleton of the non-shared-cache implementation (based
on the CS code) that could be tested.

Status: 50% done (LM only).

3) Y-valve is ported from Vulcan, DSQL is trivially moved into the
engine. At this point we have a proper architecture in place.

Status: 50% done (Y-valve only).

4) JRD is merged. We get rid of the SUPERSERVER/SUPERCLIENT/REQUESTER
conditionals. NBACKUP is carefully reviewed for compatibility with the
Vulcan's page cache.

5) DSQL is merged. Compiled statement cache is implemented.

6) INTL is adjusted to the new code.

At this point, we can start testing the embedded mode.

7) Remote subsystem is replaced with Vulcan's one.

Here we have the server mode working.

8) Utilities and services are adjusted to the new code.

Of course, some items could be moved in time, e.g. the remote subsystem
merge could be started earlier.


Your Major Question
-------------------

Why is FB used as the background for the whole tree if many of its
subsystems will be replaced with Vulcan's ones? My answer consists of
three parts:

1) Using the FB codebase, we may have the initial steps performed
without losing the stability much (and hence test the beast
incrementally), because we replace subsystems one per one. With Vulcan
being the base point, we start with bugs everywhere.

2) Number of post-Vulcan-diverge changes in FB is really large. Tracking
them one by one is not going to work and using a visual diff tool will
be much complicated for the refactored modules. In this situation,
porting the design ideas into a working code looks easier to me.

3) Unfortunately, Vulcan's code is far from looking as a production one.
Currently, it's a mixture of FB-style and Vulcan-style code that behave
differently, with some parts being commented out or totally broken. It
requires a total cleanup, reformatting, functional review and fixing. I
doubt we can afford that now.


Comments, additions, critique?


Dmitry