Friday, 31 July 2009

Re: STEP database structure

Dear STEP Team,

It is exciting to see people coming together to
make this happen. Chris, thank you for reviewing
the offerings at CrossWire in consideration for this project.

Let me start by saying the obvious: I'm
biased. So, In this email I'll only recommend
what I think should be ultimately done with
STEP. I'm more than happy to help with info and
anything else we can offer, even if decisions are
made otherwise. But here are my obviously biased opinions.

I think STEP should start with the CD CrossWire
already offers for free download which contains all of our materials.

ftp://ftp.crosswire.org/pub/sword/iso

This CD needs alot of work. First, the last cut
is from 6/2008 and quite a lot of progress has
been made here over the past year.

Second, the CD needs some congruency. It
currently just has everything we do here at
CrossWire thrown onto a CD. It would be nice to
just "do the right thing" when placed into a Mac,
or Windows, or Linux box. It currently takes
some browsing and reading of "README" files if,
for example, you'd like to install our software on a Mac.

Third, "the right thing" might be different for
STEP than for CrossWire. At CrossWire we offer
software which runs on PDAs, Mobile Phones, and
Web Servers, as well as the standard desktop systems.

Let me list some of the advantages to using CrossWire tools:

We have a common cross-platform C++ binary engine
which is at the heart of all of our tools.

This engine works the same on all platforms,
uses the same library of materials,
has a remote book installer which allows many
organizations to publish materials which will be
'auto-discovered' by any of our software,
solves the difficult if not subtle problems of
Biblical software development like,
verse reference parsing which can parse most
anything you can throw at it in 38 different languages,
dynamic versification schemes (sadly Bibles
around the world and many English Bibles do not
use the same versification scheme),
ability to research against or simply show/hide
features in Biblical text like: word lemmas and
morphology (e.g. strongs, hebrew morphemes,
etc.), footnotes, cross-references, section headings, words of Christ, etc.,
fully Unicode compliant and internationalized for
easy translation into new languages and supports
toggling (for display or search) of various
language/writing specific features like: Hebrew
Cantillation, Hebrew Vowel Points, Greek
Diacritics, Arabic Vowel Points, Arabic Shaping,
Papyri Annotation, Manuscript Variants, etc.

We have many programming language bindings to the
engine which allows developers of differing
skills to be comfortable in their preferred
language: native C++, Python, Perl, Java, most any language SWIG supports.

Native frontends have already been written and
provide a base to start. Bibletime and Xiphos
are frontends which both run on all three major
platforms: Linux, MacOSX, and Windows.

There are multiple web-based frontends already
developed and in use, including SWORDWeb and a
really cool firefox plugin done by a nameless
developer in Russia, for which I can't remember
the name of his software either, but it's cool and looks nice.

Anyway, we've had 20+ years of experience
developing software for Bible study. I think we
have a good base to offer, and I selfishly want
the STEP project to add one more frontend to
cover what you feel is missing in the free Bible
software development world. I'm sure your
contributions will benefit our large community of
developers and users, and we'd like to share in
work together with you. I'm sure you have much to teach us.

My suggestion would be to post your agenda on our
developers' forum and see what ideas you get and
who else is excited about your work.

http://www.crosswire.org/mailman/listinfo/sword-devel

There is always a learning curve to pay when
choosing to start with another codebase, but
hopefully that curve is shorter than starting
from scratch, and also you get the warm and fuzzy
feeling of contributing back to our work, as well! :)

Anyway, this is long and probably enough for now.

-Troy.

Chris Burrell wrote:
>Dear All
>Good to "virtually meet you" Troy!!!
>A few months ago, back in July/August, I was
>having a look at the Crosswire project and
>downloaded it to have a play around with it. I
>don't really know much about it, but it seems to
>have a neat Java API for use in the offline desktop applications.
>I believe the Crosswire search engine has
>already been linked to a web interface before,
>the source code of which is in the Crosswire
>subversion servers: http://www.crosswire.org/study/
>I wonder if we could get an offline version of
>this working, whether we would want to try and
>light local integration with STEP such as a web
>service of some sort (not too keen on SOAP
>myself any more, but maybe a simple REST API.
>The looser we make the coupling the easier it
>will be to upgrade the offline web
>crosswire. Obviously, we'll probably want to
>bypass the web user interface bit, but a loose
>coupling would allow us to choose our database/server structure separately.
>I believe the web SWORD project is a Java web
>app using a standard Tomcat server, although
>when I first looked their seemed to be some
>integration with COBRA-like things. Maybe we
>would use their part of the integration with
>COBRA (CORBA?) and then build our app on...
>Sorry throwing a lot of ideas out there!!!
>If we're going to interface at anyone of those
>levels it is likely our distribution will have
>to require installation of Tomcat and its configuration...
>Another option, almost completely different
>would be to take their Java Swing JSword
>installation, and branch off it (either formally
>or using a cut of their code), and build STEP
>into it. Like that we 1- benefit from a wider
>distribution range when updates are requested by
>existing users, and 2- have all the UI there
>already for the Bible searching facilities...
>Maybe introduce a few tabs. That would give us
>one complete product as opposed to a couple...
>On the other hand it ties us down to what's
>already there and maybe it would get too
>complicated (we could perhaps provide two
>tightly coupled applications - two links to the
>users to keep the UI clean and uncluttered)
>Or I guess, which having looked at the SWORD
>website, we could use the SWORD API which I
>guess is why it's there and then we'd write our own UI for it!
>I we were to go for a Java option, there are a
>number of offline pure java databases. One
>recommended to me is the one that comes with
>JBoss (haven't looked at that yet). Others like
>JavaDB might be shipped by default with Java 1.5
>I think before I can get simple pages up and
>running, it would be good to define the target
>language platform and the database. I'm pretty
>sure using Gears might be quite different to using JavaDB or MySQL.
>Having thought more about the VM idea, although
>really portable, I think it might be tricky to
>get installed all in one go and might also be quite resource intensive?
>Skill-set wise, I don't mind really, but I tend
>to be better at integration type things...
>Either way, I think we need to figure a
>high-level architecture before we start on the other things....
>Chris
>2009/7/14 David Instone-Brewer

> Dear Chris & David (& Troy)
> Troy: Please let me introduce you to Chris and David who are
> struggling, with a little input from me, to find a database engine
> for STEP.
> Chris & David: Let me introduce Troy Griffitts - he's the genius
> behind the CrossWire project, and the one who developed its
> standalone database, as well as the OSIS structure for Scripture
> texts. The datasets from Crosswire will be extremely important in
> STEP, and ideally we'd like to use the CrossWire search engine if we
> can link it to a web interface. Troy had some cool ideas about how
> to do this which, admittedly, I didn't fully understand when he
> 'explained' them to me.
> Troy: As you'll see from the messages below, we've been thinking
> about the issue of providing a database which will run on maximum
> number of platforms with minimum installation and no internet
> connection.
> We have come up with the following ideas
> * installing MySQL or something like PostgreSQL with Apache using
> something like XAMPP or EasyPHP, or on its own (see here
> <http://dev.mysql.com/doc/refman/5.1/en/windows-install-archive.html>) .
> * creating a virtual machine image containing all the software we
> need using something like the free
> <http://www.vmware.com/products/player/>VMware player
> <http://www.vmware.com/products/player/>
> * using Gears APIs
> * use JPC - a java x86 emulation which runs inside Browsers (now
> open source - see here
> <http://javafanatics.blogspot.com/2007/03/jpc-project.html>)
> I don't really know enough about the Crosswire materials to know
> what would work best.
> We don't have to use the Crosswire engine - we could just extract
> the data - but it would certainly save time to use an engine
> Crosswire has already developed.
> We don't want to access only the Crosswire materials. We will need
> to create several flat-file databases, so we'll have to think about
> whether these should be entirely separate or whether to incorporate
> them into the Crosswire engine.
> What do you think, Troy?
>
> David IB
> =============================
> Dear Chris & David
> You are both having some interesting thoughts!
> I'd like to aim for as many platforms as possible.
> Africa has more phones than PCs, and that is likely to continue.
> Phones seem to evolve and improve quicker than PCs.
> I usually lag behind with such things (I hate the hassle of being an
> early adopter),
> and yet my phone (Toshiba G910, £150) has a 800x480 screen with 16Gb
> on a microSD card,
> and I imagine this kind of thing will become more and more available.
> I suppose we will also have to bear in mind low res cut-down units
> like iPhones
> which only have 320x320 resolution (I think) but I'm sure this will
> improve too.
> We don't want to keep up to date with lots of operating systems,
> especially things like iPhones which are mostly locked down,
> so we want to be as standard as possible.
> I had a quick look at the Google Gears page and saw something which
> heartened me:
> Compatible with: * Windows
> XP/Vista * Firefox 1.5+ and
> Internet Explorer 6.0+ * Linux
> (details) * Firefox
> 1.5+ * 32-bit OS (64-bit not
> supported) * Mac OS X 10.4+,
> G4+/Intel CPU * Firefox
> 1.5+ * Mac OS X Leopard 10.5.3+ or
> Tiger 10.4.11+, G4+/Intel CPU *
> Safari 3.1.1+ * Windows Mobile 5+
> (details) * Internet Explorer
> Mobile 4.01+ * Windows Mobile 6
> touchscreen devices (details) *
> Opera Mobile 9.51 (Presto build
> 2.1.0+) * Android 1.0 device or
> emulator Gears is available for
> Windows, Windows Mobile (IE Mobile, Opera
> Mobile), Mac (Firefox, Safari), Linux and Android.
> David IB
> > I know there are many portable DBs around. I haven't used many
> myself, and
> > can't remember any names. The two databases that I know are quite
> popular
> > with PHP are MySQL and PostgreSQL.
> >
> > I know MySQL can be installed without the installer on its own:
> >
> http://dev.mysql.com/doc/refman/5.1/en/windows-install-archive.html
> A quick
> > look at that guide suggests, to me, that it should be fairly easy
> to install
> > it in a custom automated way. (but that was just a glance).
> >
> > I'll have a think and let you know.
> > Chris
>
> At 09:21 14/07/2009, Chris Burrell wrote:
>> A guess a few other questions would be...
>>
>> How much information is stored locally? The whole thing? just a
>> little bit? Do we need offline syncing facilities? Do we require
>> any web-based database at all?
>> Would a database supporting replication be useful for doing this?
>>
>>
>> Chris
>>
>>
>> Sounds like a good idea...
>> What are our target
>> platforms? PCs/Macs, with plenty of
>> space? Power enough to run VMs as well
>> as the host operating system? Mobile
>> devices maybe? Local storage on local devices?
>> Those are only a few of the questions. If we're talking
>> apache/mysql, we also need to figure the best browsers and
>> what to support... There may well me some good php/java
>> framework like cakephp, jsp type stuff which will make the
>> development a lot faster...
>> Just wondering what the criteria might be to decide on an
>> architecture
>> Chris
>> 2009/7/13 David Hollands <admin@remza.co.uk
>> <mailto:admin@remza.co.uk>>
>>
>> Hey guys,
>> An alternative would be to preconfigure a virtual machine
>> image containing all
>> the software we need. The only prerequisite
>> is a copy of the free VMware player (
>>
>>http://www.vmware.com/products/player/
>>) which we could distribute on the usb stick along with the
>> image. The image would
>> be a small open-source linux operating system
>> with some scripts to
>> start the servers/browser etc. We could use the
>> Browser Appliance as a basis (
>> http://www.vmware.com/appliances/directory/80 ).
>> David.
>> > Hey
>> there > > I know there
>> are many portable DBs around. I haven't
>> used many myself,
>> and > can't remember any names. The two databases that I know
>> are quite popular >
>> with PHP are MySQL and
>> PostgreSQL. > > I know
>> MySQL can be installed without the installer on
>> its own: >
>>
>>http://dev.mysql.com/doc/refman/5.1/en/windows-install-archive.html
>> A quick > look at that
>> guide suggests, to me, that it should be
>> fairly easy to
>> install > it in a custom automated
>> way. (but that was just a
>> glance). > > I'll have
>> a think and let you know. >
>> Chris > > 2009/7/13 David Instone-Brewer <

>> >> >> Dear
>>Chris >> >> I want to
>>get down to looking at your skills next to
>> the needs of STEP >>
>> this week. >> It looks like you may be the ideal guy for making the
>> html or php pages >>
>> which define the basic functionality of the
>> structure. >> >> The
>> big problem at this moment is that we still haven't
>> worked out what >>
>> database to use! >> We don't want
>> to use MySQL (my default for this kind of
>> project) because >> we
>> want to make it workable without an internet
>> connection, and MySQL
>> would >> require installing a MySQL server. This is easy with
>> something like
>> EasyPHP >> or XAMPP, but this would be more installation than we
>> want, and it
>> wouldn't >> be cross-platform. David Hollands <admin@remza.co.uk
>> <mailto:admin@remza.co.uk>> is
>> looking into >> GoogleGears. If
>> you have any ideas on this subject, we
>> would both be >>
>> interested to hear. >> >> David IB

No comments:

Post a Comment