Bio Rails Development

More 2.3.2 Upgrade issues

2009-04-09T23:21:00.003+01:00

Lots of ActiveRecord::StaleObjectError errors

Ok took some time clearing these. The basic problem was with large complex trees of objects resulting in multiple paths destroying the same objects. For example a 3 level process <-- context <-- parameters. As a handy shortcut process has a collection of parameters and contexts. The delete was cascading down via both process->context->parameter and process->parameter. This resulted in a attempt to delete a record twice.

Methods of fixing these problems:-

look carefully and change see if :delete_all raw delete can be used for dependent records.
look a collection and see if :destroy options can be removed
monkey patch to activerecord/lib/active_record/locking/optimistic.rb line 126 so only errors if real problem with stale record, allow duplicate deletes.

My little patch of Active Record to the last few problems as the following:-

unless affected_rows == 1
raise ActiveRecord::StaleObjectError, "Attempted to delete a stale object"
end

Changed to add test for already delete record and say which object actually had the problem :-

if affected_rows != 1 and self.class.exists?(self.id)
raise ActiveRecord::StaleObjectError, "Attempted to delete a stale object #{self.class}[#{self.id}]"
end

Rails 2.0 -> 2.3.2 Upgrade

2009-04-01T09:49:00.002+01:00

General the upgrade was not a major problems most of the application work. Have about 25 or 2500 tests failing, mainly with stale record on delete for lots of controller and unit tests of destroying object. The main issues where as follows:-

Cleanup
Had a few problems with plugins causing problems after update of rails:-

plugin/engines needed to be deleted,now part of rails
plugin/arts needed to be deleted

Also had a few renames of Dependencies.load_paths to ActiveSupport::Dependencies.load_paths from the 2.1+ days to sort out

Testing
This caused a number of problem due to widespread use of the Test::Unit::TestCase in the code. These had to be changed a number of large search and replaces in the test tree.

unit tests replace Test::Unit::TestCase with ActiveSupport::TestCase
function tests Test::Unit::TestCase with ActiveController::TestCase

Database Migrations
Needed a little hack to move development database to rails 2.3.2. Needed to fill the schema_migrations with old style version numbers as follows:-

1.upto(391){|i|DataConcept.connection.execute "insert into schema_migrations (version) values(#{i})"}

Todo List

Need to work on destroy test failures, to see if bad code or test under 2.3.2
gloc plugin does support new internationalization API will have to be changed
Need to test modified action web services plugin with 2.3.2
Need to check with ruby 1.9.1

Starting on Biorails 4.0

2009-04-01T09:41:00.003+01:00

Boy been a busy year of delivery and going into production on a number of sites. Have not had time to write for ages. Now starting the majors upgrades to Biorails 4.0 cycle. Currently looking and ruby 1.9.1 and rails 2.3.2 updates.

First three rails things todo are:-

1) Update to Rails 2.3
2) Update Action Web services to work
3) Update Oracle drivers to support bulk inserts
4) Check gems on ruby 1.9.1

On the application side

1) Redo the unstructured data management. Time to make the product a better authoring tools
2) Support a lot better printable output in Open Office,word and PDF formats
3) Enhance the recipes to allow creation of a multiple level tree of items from a recipe
4) Time for UI make over again

Synergy on ubuntu 8.04 , freeze up fixed.

2008-10-31T10:12:00.007Z

Synergy has been my long term tool for sharing keyboard and mouse between my development and build machines. Unluckily with ubuntu 8.04 there has been a lots of 4-5sec freeze up problems. In the end moved to 2x keyboards and mice again.

Now this is fixed with some open googling and some source patching, as follows.

1) download 1.3.1 source code from http://synergy2.sourceforge.net/
See http://downloads.sourceforge.net/synergy2/synergy-1.3.1.tar.gz

tar -xzf synergy-1.3.1.tar.gz

2) Patch with fix from lanchpad

See. https://bugs.launchpad.net/ubuntu/+source/linux/+bug/194029

wget http://launchpadlibrarian.net/13995863/synergy_diffs
patch -p1 < synergy_diffs

3) Install needed extra libraries (XTest)

sudo apt-get install libxtst-dev

4) rebuild and install

cd synergy-1.3.1
./configure -prefix=/usr
make
sudo make install

Restart synergy all works smoothly again. Can get space for coffee cups back again.

Note: Plus 12hours some problems with synergy going wild and eating 100% cpu so not a full solution looks like I have to review whats happening. Not much betters solution the the run with sudo method I used before.

Biorails 3.0

2008-07-01T12:01:00.003+01:00

Ok have merged and release 3.0 code onto core. The http://biorails.org/svn/biorails/core/trunk contains 3.0 code release now. All that is needed is lots of web site updates and packaged installations of open source version. Afraid followed the money can concentrated on enterprise version for last few months.

Have done packaged windows and linux installation of biorails for enterprise version. Now need to do packaged open version on windows for people to play with.

Getting to the full up in 5 minutes up and working experience is a hard for a web application. Almost there now with windows thanks to sqlite as a database. So think first release with a standalone demo version on windows.
Some more work needed for linux versions as apache front end still needs to be hand edited.

Oracle 11g /Ubuntu update

2008-05-31T23:02:00.004+01:00

Ok found the magic option to speed up oracle 11g for my development box.

emctl stop dbconsole

Now converted back to simply database server its works a lot better. Afraid after some time looking a performance stats found could not find my application in top SQL . The enterprise console performance stats only showed enterprise console eating all the resources.

Database on its own appears to be comparable but a little larger then XE. Afraid think in case of oracle, enterprise may still mean crew of 100's and something the site of a aircraft carrier to run it at full potential. But that what its aimed at the 1Tb+ database not developer/test systems.

Think I may go back to XE one tested everything works with 11g as expected.
Fun next as time to see how stacks up against Sqlite, Mysql and Postgres in running the all the my test suites.

Oracle 11g on Ubuntu 8.04 64 bit / Life on the Edge

2008-05-29T22:39:00.008+01:00

Decided to experiment with oracle and taken the hard road with Oracle 11g on Ubuntu 8.04. This was is my main development box and a good solid dual core AMD64 box. Just upgrade from 6.06 to 8.04 ubuntu and its time to give 64 bit oracle another go. Last time gave up and when 32 bit of OS and oracle, with a simple Oracle XE installation.

Before started found the following articles to lead the way

Oracle 10gR2 on Ubuntu 6.10 AMD64

The blog by Halis way was a great help as really acted as a template for my 11g my work. As other have notedf one of the main catches is that the Oracle Universal Installer (OUI).
So we need a 32-bit Java triggered and a soem 386 compatability libraries to get it working.

There were some linkage problems with the 32bit exproc which is used with C++ database procedures and ultra text search. I simply continued through this as not planning to use exproc. I still have to solve this problem so remember this a play installation oracle and not suitable for production/heavy use. If to really want a supportable oracle platform afraid best to swap to openSUSE or CentOS.

Machine Setup
First of all make sure your machine set up as a server with a static IP address and host name pointing to the external ip address. Check or edit /etc/network/interfaces to make sure the interface is static. For example my interface look as follows:-

auto lo
iface lo inet loopback

iface eth1 inet static
address 192.168.1.10
netmask 255.255.255.0
gateway 192.168.1.1
auto eth1

iface eth0 inet dhcp

On my box eth0 points out to the world and eth1 is inhouse network. Then double check the /etc/hosts to make sure there a line for the server name

$ grep $(hostname) /etc/hosts
192.168.1.10 acles64

If you see 127.0.0.1 just edit the hosts file to have the external IP.

Now its time for a lot of compat libraries and 32-bit libraries.

sudo apt-get install build-essential zsh libaio1 lesstif2
sudo apt-get install lesstif2-dev make libc6 libstdc++5
sudo apt-get install lib32stdc++6 lib32z1 ia32-libs

This will probably cascade in to quite a few packages depending on how much you have installed, be patient and let it download everything it needs. The extra ia32 libraries is probably the biggest change to the old 32-bit installation.

Edit /etc/sysctl.conf to have the kernel parameters required by Oracle
(Afraid have not put any thought here simply copied Halis way 10g ones, may not be ideal for 11g or needed).

Add these lines at bottom of/etc/sysctl.conf :

# Oracle stuff
kernel.shmall = 2097152
kernel.shmmax = 2147483648
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default = 262144
net.core.rmem_max = 262144
net.core.wmem_default = 262144
net.core.wmem_max = 262144
vm.swappiness=10

Create the oracle user and required groups.

sudo groupadd nobody
sudo groupadd oinstall
sudo groupadd dba
sudo useradd -s /bin/bash -g oinstall -G dba oracle
sudo passwd oracle

Increase system limits by adding these lines to /etc/security/limits.conf:

soft nproc 2047
hard nproc 16384
soft nofile 1024
hard nofile 65536

Next added a few symlinks to avoid any script problems:

sudo ln -s /usr/bin/awk /bin/awk
sudo ln -s /bin/true /bin/rpm
sudo ln -s /usr/bin/basename /bin/basename
sudo ln -s /lib/libgcc_s.so.1 /lib/libgcc_s.so

Oracle installation
This was done under the oracle user and dba group was added for access to oracle. I like /opt/oracle as a installation route and luckily had space there.

sudo mkdir -p /opt/oracle
sudo chown -R oracle:oinstall /opt/oracle
sudo chmod -R 775 /opt/oracle

I downloaded Oracle 11g and unzip its to /opt/oracle/database. Then run the installer, make sure you are allowed to use the X11 server.

May need to login as oracle or play with export DISPLAY, xhost, vnc etc to get it working.
To get the 32-bit JRE to work you need to set the XLOCALELIBDIR before running the installer:

export XLOCALELIBDIR=/usr/lib32/X11/locale

cd /opt/oracle/database
./runInstaller -ignoreSysPrereqs

I just ran the defaults through with following corrections

1) Set location to /opt/oracle/product/11.1.0/db
2) Corrected oraInventory location as /opt/oracle/oraInventory
3) Continued with two linkage errors

The linkage errors were luckily in non vital areas (free text search) and (exproc32) so contained and have got my biorails application installed. The Error are shown below:-

Linkage of exproc32

INFO: End output from spawned process.
INFO: ----------------------------------
INFO: Exception thrown from action: make
Exception Name: MakefileException
Exception String: Error in invoking target 'all_no_orcl' of makefile '/opt/oracle/product/11.1.0/db/rdbms/l$
Exception Severity: 1
INFO: Calling Action unixActions10.2.0.1.0 make
registerOnly = false
installMakePath = /usr/bin/make
installMakeFileName = /opt/oracle/product/11.1.0/db/srvm/lib/ins_srvm.mk
installTarget = install
undoMakeFileName =
installArguments = ORACLE_HOME=/opt/oracle/product/11.1.0/db
logFile = /opt/oracle/product/11.1.0/db/install/make.log
undoTarget =
progMsg = Linking RDBMS Executables

Then Later there a problem with the Oracle Text INSO filter ctxhx needed by Ultra Search crawler . As not planning to use not worried at present.

INFO: make: *** [ctxhx] Error 1

INFO: End output from spawned process.
INFO: ----------------------------------
INFO: Exception thrown from action: make
Exception Name: MakefileException
Exception String: Error in invoking target 'install' of makefile '/opt/oracle/product/11.1.0/db/ctx/lib/ins$
Exception Severity: 1
INFO: Calling Action unixActions10.2.0.1.0 make
registerOnly = false
installMakePath = /usr/bin/make
installMakeFileName = /opt/oracle/product/11.1.0/db/rdbms/lib/ins_rdbms.mk
installTarget = ipc_g
undoMakeFileName =
installArguments = ORACLE_HOME=/opt/oracle/product/11.1.0/db
logFile = /opt/oracle/product/11.1.0/db/install/make.log
undoTarget =
progMsg = Linking UDP

Once this was done could go in and create users, roles, tables and schema I needed and appears to be working fine. Only real problem its eaten 1Gb of memory and taken over my development box in standard oracle manor. Generally think Oracle XE makes the better developer platform as a lot lighter on resources.

Went in and cut things back a lots and got memory back for my development IDE.

Is Windows Vista a lost cause?

2007-12-23T21:21:00.000Z

Ok been, started asking around and cant find a single person I know say they prefer vista vs. XP. Even the die hard .Net developer I know think XP is better. Dont understand this, there was supposed to be loads of usability studies etc. What happened someone forgot to include anyone who actually uses a computer full time or previous version of windows in this.

I really loved the original windows NT when it came out, I purchased a Windows Tablet as loved the idea and still use it a browser to this day. As a developer I like Windows for for rich client stuff, spent years in visual studio building application. On server side used to use NT everywhere but have moved to Linux in recent years. But now appear that windows may have managed to shot itself in the foot on client side. I'm developing a growing dislike for vista with each time I try and use it. Just don't feel in control, cant work out why things like internet access aren't working, its now getting to stage of not wanting to see it again. I keep going backing thinking, it must be me, but then give up and reboot in Linux to actually have get the machine to work well.

Wireless, Media playing, simple internet browsing all work better on the Linux side and whole machine just feels so much more snappy. Only thing that works better on windows is hibernate and suspend which are really poor on the linux side. But happy to boot and shutdown a machine to have a working systems. Doesn't hunt that actually boot and shuts down fast don't really care.

Windows Vista v.s. Mint Linux

2007-12-19T11:48:00.000Z

Ok got a new laptop for traveling yesterday, a little Toshiba R500 <1kg job, so got first real exposure to vista. Afraid as after function not glitter so cant really see much in new Aero UI. Once turned off all the pointless glitter system as a lot more responsive and was happier may be better then XP but still liked the simple 2000 interface better . Cant really see a reason to upgrade any existing systems, as appears the same odd system with a repaint and new dice on the mirror.
May be a little harder to find stuff, but this is simply due to moving things around. With each new version the bits I use as developer simply appear to move down a level and become a little harder to find (Environment variable, services, folders in file manager etc).

Had to spend a few hours watching it configure, update and reboot its self a number of time as found and installed my tool set, SqlYog, Enterprise Architect, NetBeans, Firefox, Skype, Mysql etc. All worked got a little board of now going to do xxx dialogs. All worked, but cant understand were the 780 mb or rams gone before installing any applications!. End results 4 hours work to get to a usable state and most of my environment build.

Finally got bored to put in Mint 4.0, my main desktop,and installed. The difference was shocking 10min later was installed and working . A simple selection package later and was up in working. As use crossover even the windows application installations worked faster and easer then under windows. Basically about 20 minutes later was up and running. Only real difference it computer simply appeared more responsive and faster.

End result it have not rebooted into vista again, will keep it around in-case get a nice .net C# project but otherwise Mint got the edge for my usage and just makes the computer appear faster!
Afraid believe role of a operating system is to run application for me not use all the processing cycles its self. Afraid vista appears to forget this, Linux remembers.

Oracle and Rails 2.0

2007-12-18T00:34:00.000Z

Oh well once you think you done, alongs coming testing in a production environment and some extra fun. Turns out no one else appears to be working with Oracle and Rails 2.0 yet.
No one implemented the needed select_rows in svn. Caused a few problems to use the driver in tsting.

In put ended up with a nice custom oracle driver in the end. Need a few minor changes:-

Modified mapping to my own likes

float <-> binary_double
char(1) N/Y <-> boolean
text <-> varchar2(2024)

Boolean (true t 1 y +) all mapped to true
Implemented select_rows
added a high/low sequence which I prefer for high insert rates with oracle

the current version of the gem I created is up at http://biorails.org/svn/biorails/contrib/activerecord-oracle-adapter-1.0.0.gem if anyone interested. At present its a work in progress. Need to put on my old oracle DBA hat and look for anyway to improve it. But may help anyone else moving up to 2.0 with a oracle backend in the next few weeks. Would like to use Insert returning for sequences and trigger filled values if possible but will have to have a further look.

Put needed changes on ticket: #10415 in rails dev.

Finished migrating to Rails 2.0

2007-12-15T15:44:00.000Z

Just a not to say code base is now passing all test with Rails 2.0. Took about 8 hours of effort in the end mostly with scope and SOAP changes.

So feel free to upgrade Biorails and try it out and upgrade local rails if you want. At present environment.rb does not contain a version to will attempt to run on most recent installed rails version.

Expected status:-

Units: 353 tests, 668 assertions, 0 failures, 0 errors
Functionals: 301 tests, 780 assertions, 0 failures, 0 errors

Have got a simple does it return correctly or throw a exception test on most controller actions now . Only real problem is create/update functional are not really testing all the create/update objects with generation of new records.

Rails 2.0 upgrade produced a lot of donkey work but went smoothly the mains changes needed are shown below:-

1) Paginate.new => model.paginate via will_paginate plugin

Keep a eye of bottom of pages to check new paginate control is coming up ok in visual testing. Buts appears ok on initial pass

2) end_form*

Have had to change a number of forms to block wrappers to

<% form_tag( stuffff ) do %>
<% end %>

3) with_scope

Use this a lot to manage visibility of data. Has gone protected so having to move filters to part of correct model. Model.with_scope => self.with_scope and code moves into correct models and referenced so

On the whole appears a litte faster other then this no real problem seen.

def self.visible(*args)
self.with_scope( :find => {
:conditions=> ['exists (select 1 from memberships m,studies s where m.user_id=? and s.id=study_protocols.study_id and m.project_id=s.project_id)',User.current.id]
}) do
self.find(*args)
end
end

4) Had to Add 3 plugins to externals list in svn to cover bits moved out of core rails:-

script/plugin in_place_editing
script/plugin act_as_tree
script/plugin will_paginate

Need to find if there a Oracle plugin.There is talk that oracle ActiveRecord has moved out to separate gem/plugin could not find this yet. Will have another look later but will not effect development or testing yet.

5) Acton Web Services

Well this was a pain, but changed over to a plugin based on 1.2.6 code under my control at present. Will have to test this with our .net office client to make sure its working well.

ActionWebService and Rails 2.0

2007-12-15T15:24:00.000Z

Well looks like I have to take on supporting this for my BioRails project. Afraid on the client side we are making heavy use of .NET generated client code generated from the WSDL. This is used for the Microsoft office client integration is isn't going to become restful soon. Also have clients using the same historic interface so ended up picking up AWS porting to 2.0.

To install the plugin the following should work:-

script/server plugin http://biorails.org/svn/biorails/plugins/action_web_services

At present all I can say is it passes our in-house test suite for the web services I want. Beyond that I have not had time to go. Had to do a few minor changes to get invoke to work, fully made it display calls sorted by name which have been meaning to do for ages.

Maybe for online services could keep few mongrels up with 1.2.6 dedicated to web service requests after move to 2.0. But as we distributing a software package instead of a service this would be hard on customers.

Basic changes are setting the gem dependencies to allow for use with Rails 2.0

gem 'activesupport', '>= 1.0.2'
gem 'actionpack', '>= 1.6.0'
gem 'activerecord', '>= 1.9.0'

Then the removal for include Reloadable::Deprecated from the code base. This appears to have worked for me. I expect there to be a similar gem soon. Then a number of conversions of render_file to render :text => content in the scaffold code.

Its now back to working on 2.0.1 with my .net client usage pattern. In fact working really well on client side as collections of simple data objects is a common pattern for .net and works well for building the client UI.

Web vs. Application development

2007-12-11T21:35:00.000Z

Have noticed Im have come full circle. Im not writting a web application any more in Ruby on Rails. I am builder a n-tier rich client application, the difference is the client is javascript+html in a web browser as a container. I now finding the MVC part moving to javascript with rails supplying the initial page template and then a few restful json data structures to handle change in state. Affraid cant see the point in the X part of ajax when talking from ruby to javascript its a classic cut out the middle man xml. The .xml there on the action but .to_json faster then .to_xml for moving the data.

I have been reading the ajax is evil etc talk, and this may be true to public wildly distributed view anywhere application. But I'm after the best user experience for a known set of clients for in-house usage. For my style of application JavaScript and Ajax is a boom as take away the web bar at the top with Mozilla prism and user just think its a richer client application like the host of .net application. In fact in user pilot beat a competitor thick client swing application as user rated the web page as having better windows integration (odd but true).

Building the UI in Javascript

2007-12-05T12:58:00.000Z

Its turning point time with release of ExtJs 2.0 My web forms are now looking richer and better then in then in previous generation of applications with thick client swing UI's. Ok its not as easy as working visual studio and .net, this still has the edge, but feel its at the same level now as old style hand crafted swing.

Have Tree, live feed combos ,drag and drop, data grids, validation and dynamic form editing sorted no. Can build and run all my major data entry UI in rich combination of rails and extjs.
Only down side its about 500 line of javascript for a complex UI which dwarfs the initial ruby implementations. This may not have the UI simplicity loved by the a lot of core rails people. But provides the richness expected on the thick client crowd without the pain of distribution and installation of 1000's of machines.

All that really lacking is solid graphics. Would really love all browser to support svg as think best approach would be a mixture of ajax and vector definitions from server. Could go the flash way for this and may well, but like open standard approach if possible.

Back to Ext Javascript

2007-09-27T23:56:00.000+01:00

Back to UI enhancement today putting reports results into a grid in a flexible manor. Big debate is whether to go for a html approach and use javascript as conversion of the presention, or drop presentation and go for JSON data hash to client and do the full presentation there.

HTML has better fallback to hmtl but JSON as less traffic and server overhead. Currently prototyping the JSON route. But a close call really.

How to you pick a database.

2007-09-26T22:12:00.000+01:00

How to you pick a database. Normal answer is the customer desides for you if is a large project with a must run on X.

If given a free choice the decision goes like this.

Database size > x 100 server memory given me Oracle any day.
Database size < x 10 server memory any big name database will work well
Database <= server memory would use postgres as default , using it on off for years, its does everything a rdms should.

When your having rows appears in 10M blocks oracle partitioning and other trick are a god send. Otherwize most of the database out they do eventhing needed.

In the old 2 tier client/server days, always went for oracle because of Pl/Sql and used the oralce as combined application server and database with all business logical in the database. Now with n-tier only really want the database to do SQL database stuff to dont care so much. In porting applications arround the playing field very level for both open source and commerical offerings. All major ones to joins,foreign keys, unions etc.

Process v.s. Productivity

2007-09-26T22:08:00.000+01:00

Spent the day replying to a RFP.
Client done a great job of written the RFP. Only real worry is with the amount of work in the document and our reply may actually be the same as actually doing the project in Rails.