3-in-1 : Merging bibliographic data

iFind Discover has a single bibliographic database extracted from the three partner libraries. It was decided early on that we wanted to de-duplicate records as much as reasonably possible, so records were extracted from the local LMSs and then pre-processed in order to:-

Remove duplicates. At present, this is done solely on ISBN/ISSN, but this creates problems, e.g. where serial titles have changed but retain the same ISSN. I am working on a more sophisticated merger process. When records are merged, a preferred record is selected and then protected fields (e.g. local notes, online links) are transferred from the non-preferred record.

• Store system numbers. VuFind normally uses the system number in the 001 field to connect to the LMS in order to obtain holdings information. Since 001 is a non-repeatable field, part of the changes we have had to make to VuFind is to use another field to contain links to the LMS. We chose field 969, for no good reason other than it is a local field and not used in any of our existing records. The field contains the institution’s initials in subfield a and the system number in subfield b.

• Replace the record system number with the ISBN/ISSN, which is used to match duplicates. 10-digit ISBNs are converted to their 13-digit form.

• Transfer holdings data. Where holdings records are included in the extract (currently only at Swansea University), the 852 and 866 fields are merged into their parent bibliographic record. This makes the shelfmarks searchable even if they were not present in the bib record, and it allows for location information to be displayed even when the LMS is offline.

• Correct oddities. There are some UTF8 character set problems and have been other structural problems in one of the datasets, to which the Perl MARC modules took exception (although these have mostly been fixed at source now).

At present, the pre-process and load procedure is manual and being performed roughly once a week, but we are working on an automated procedure for daily updates. The manual procedure pre-processes and loads records from each institution in turn, and finds duplicates by querying VuFind for records aleady loaded from other institutions.

Paul

How do you solve a problem like...

One of the most common initial comments from early adopters of iFind Discover is that they didn't see the benefit of having results from other institutions because they "only cared about items in their own libraries". They also expressed concerns that searches would take longer and explained that it was more difficult to determine quickly if items were available in their own library.

The issue of performance is something that we are only now beginning to look into as it has been impossible to determine where potential bottlenecks and problems might be with a limited number of people using the service. Searching three remote catalogues is bound to effect search time so we are keen to do some benchmarking to compare out installation with similar size single LMS implementations.

We initially experienced very poor performance during peak times which saw both our CPU and Memory allocation quick consumed. Luckily, as we running VuFind on a virtual server we were quickly able to add an extra CPU and double the system memory and things have been a lot more stable since.

We are still hitting peaks of 90% CPU usage during training sessions so we will continue to monitor the situation very closely. Possible culprits for these spikes are a) The use of Jangle during login b) The login workflow in general c) Server / client settings.

We took the comments from users regarding increased difficultly in getting item information very seriously and have tried to indicate exactly which institution availability information is coming from. The planned introduction of a new tab interface for holdings which clearly differentiates between each institution is on hold due to an Internet Explorer 7 bug.

Whilst we can take care of the ease of access to relevant information, the actual benefits of a shared catalogue will only be apparent if institutions enthusiastically agree to exchange resources and promote the service.

Luke

JISC LMS Programme Meeting at ECDL 2010

Our three project members, Mark Hughes, Paul Johnson and Luke O'Sullivan attended the JISC LMS Programme Meeting at the ECDL 2010 conference in Glasgow in the second week of September.

The LMS, OSS and UI strands were extremely useful and it was great to meet people with similar projects, particularly when they were willing to share ideas and code!

Many of us came to the conclusion that we were desperately trying to re-invent the wheel when we should be pooling resources to create killer applications. Once the gripes over the current market were discussed for cathartic therapy, there was plenty of optimism for the future.

From an iFind Discover perspective, we hope to benefit from the work currently under development at LSE regarding dynamic location mapping, Sussex's CREDAUL project and Kent University's Project Keven.

VuFind 2.0 Conference - Was a Multi-LMS implementation of interest?

My primary aims in attending the VuFind 2.0 conference were 1) to learn from other implementations of VuFind and apply best practice to iFind Discover 2) to take advantage of the assembled programming talent to solve problems and improve performance 3) to learn more about where VuFind was headed 4) to gauge the potential 'market' for multi-lms implementations of VuFind and 5) to try and get my VuFind developments into the trunk so that they could be improved and supported by the community, thus increasing the resilience and sustainability of the iFind Discover project. Here are my thoughts on how each aim eventually panned out:

To learn from other implementations of VuFind and apply best practice to iFind Discover
One of the great things about VuFind is that it is incredibly customisable - this allows each institution to really create a discovery platform which matches its customer profile. Of particular interest to me was the consortia code developed by CJ O'Hara, Keith Dedman, Mark Noble at opac.marmot.org, the Bookbag functionality developed by the team at York University in Canada and a similar implementation by West Michigan University. Beyond VuFind but intimately related to the discovery process, I was also amazed by the work on concordances developed by Eric Lease-Morgan and the Fascinator project demonstrated by Greg Pendlebury. Unfortunately, this customisable strength is also one of VuFind's greatest weaknesses - in essence, every installation of VuFind has the possibility of being a standalone product which makes upgrading to the latest trunk extremely problematic.

To take advantage of the assembled programming talent to solve problems and improve performance
Thankfully, everyone on the conference was very sociable and keen to exchange ideas. We were able to discuss everything from user interface design to useful programming tools and improving system performance. From this viewpoint alone, the conference was a complete success.

To learn more about where VuFind was headed
It was great to learn that Joe Lucia and the VuFind team are determined to manage the project in a professional manner and that they have begun to develop processes which will make VuFind upgrades and developments less 'ad-hoc' and model focused. There was an incredible session on the last afternoon where all the desirable additions and developments were thrown into the ring, debated and discussed and assigned to specific strands and workflows. The VuFind 2.0 roadmap is the outcome of this session.

To gauge the potential 'market' for multi-lms implementations of VuFind
After talking to delegates, it seems that the major focus (at least in the US market) is not for a multi-lms approach to the discovery layer. This obviously has massive implications for the work that I have been doing and for the possibility of getting my code into the trunk. From what I could ascertain, there is a far stronger momentum in both the academic and public sector for implementations of a single joint open source consortia lms. Evergreen is the buzz word and it seems that VuFind hopes to present itself as its OPAC of choice. The code developed by Marmot may therefore become crucial to the success of VuFind.

To try and get my VuFind developments into the trunk
Given the market focus on consortia, this seems unlikely though my code is being looked at by some interested parties. Having met such talented people at the conference, I was also made more aware of the fact that as a "Shambrarian" and a "masher" rather than a skilled librarian or developer, that my code would need a lot of TLC by some professionals in order to get it up to scratch.

Luke

VuFinding (Across the Universe)

In September 2010, I was lucky enough to attend the VuFind 2.0 Conference in Villanova, PA, USA in order to present the results of out multi-LMS work. In addition to finally meeting some of the people I owe so much to for help with iFind Discover, I was also able to look at best practice reports from a whole host of institutions from across the USA and Canada and as far afield as Australia. There are some incredibly talented people working on the VuFind project and I am pleased to report that some great developments are planned for VuFind 2.0.

The project is being given direction by Joe Lucia @ Villanova and has some serious developer talent led by Damien Katz.

Details of the conference can be viewed at http://vufind.org/wiki/vufind_2.0_conference and my presentation can be seen here

On October 8, 2010 Joe released VuFind 2.0's Organizational & Software Roadmap which can be viewed at http://vufind.org/docs/roadmap2.pdf

Luke

Why and what is SWWHEPSRCH? A brief "pitch"

At the recent JISC LMS event, we were asked to produce a one-minute pitch of our projects, so here goes for #swwhepsrch

What is swwhepsrch?

This project is a case study on the implementation of VuFind as a shared resource discovery platform for the three university libraries in the South West Wales Higher Education Partnership (SWWHEP) : Swansea University, Swansea Metropolitan University and Trinity University College, Carmarthen (now part of the University of Wales Trinity St.David).

Why a shared discovery platform?

The aim of SWWHEP is to establish greater collaboration and cooperation between the member institutions, in order to improve and expand services available to their staff and students in an efficient and cost-effective manner. For the SWWHEP library project, this involves providing users with greater and easier access to resources across the three institutions, through various means including a common RFID self-service environment, a document delivery service between institutions and full reciprocal borrowing.

To facilitate common access to resources, we need a common discovery service. To improve upon the discovery services already available, we need a next-generation interface. After examining various options, the project decided to implement the discovery interface using the Open Source VuFind product, backed up with the propriortary Metalib/SFX product for searching electronic resources.

Why VuFind and Why OpenSource?

VuFind provided most of the next-generation functionality we were looking for, and in our opinion stood up well against the commercial options. Being an open source product, there was no purchase cost, which was important given our fixed budget. More importantly, it gave us the ability to tailor it to our needs as much as we needed or desired. The most important part aspect of this was enhancing the product so that it communicates with the institutions’ three separate library management systems simultaneously, since it is the LMSs which provide live availability data and allow users to renew materials, place reservations and check their library accounts. The flexibility offered by open source also allowed us to make other changes to the interface in order to provide a library catalogue that is significantly better than any of those previously on offer.

The choice of VuFind also gave the university libraries an opporunity to try open source on a relatively small scale as a possible prelude to considering it for other applications, possibly even a full library management system.

Where next?

The new catalogue – branded iFind Discover – is now in beta test and we will be monitoring and reporting on user reaction. First-line support for the product has been transferred from the SWWHEP project to library IT staff, but there will continue to be a program of enhancements implemented on a monthly basis.

Luke O’Sullivan, who carried out most of the development work, recently attended the VuFind developer’s conference at Villanova University in the USA, and we are hopeful that some of his work, particularly regarding the multi-system connection, will become part of the main product and readily available to other library consortia across the world.

iFind Discover in "live Beta"

iFind Discover went into “live beta” at Swansea University last week! It now appears in place of Voyager on library OPACs and links from the library home page, although there is still a link back to the “classic catalogue” for those who want it.

There has been a good flow of feedback – mostly positive and constructive – and Luke has been dealing with issues as they arise. We are in the process of doing a new full dataload, with tweaked parameters, which should resolve some questions from users.

Although the SWWHEP project team is still working on improvements and fixes, first-line support is now being handled by Swansea University’s library systems team.

Paul