Best practice recommendations

 Best practice is dynamic in nature - what will be best practice now may be, in time, superseded by: innovations in practice, changes in technology, changes in law or governance structures, and expectations, values, knowledge or other influences that make methods outmoded or less relevant, useful or appropriate. For these reasons, best practice must be subject to constant review [4].

 
The experience accrued during the project, and a mix of various inputs from related initiatives and projects, the combination of different practices contributed to the provision of best practices guidelines and recommendations regarding mEducator metadata scheme, as outlined in section 3.2. More specifically, the emergence of the aforementioned best practices is based not only on successful implementation of the metadata schema, but also on mistakes and failures during its design and development, as well as on feedback from external reviewers during its evaluation.
In designing the metadata schema we have considered the following factors:
• Building on reference models and existing standards to ensure maximum interoperability in the form of the schema
• Recognizing the need to design a schema that will work with the functionality of our solutions in order to facilitate sharing, search and retrieval as well as repurposing content.
• Recognizing the need to expose and share metadata in a standard format for re-using elsewhere.
All of the above resulted in the choice of W3C RDF as the standard data model and the adoption of Linked Data principles. The schema design is grounded in state of the art semantic technologies such as RDFS, OWL and SKOS. Our experience has shown that a few iterations are normally required to find an appropriate balance between best practice and pragmatic implementation decisions and the level of detail required in the schema [24].  Therefore, taking into account the above factors, as well as the evaluators’ feedback on the schema and looking also into adopted approaches adopted in similar work, we can distil our experience to derive the following points:
 
There is no metadata schema that is perfect and fits all needs. This is being increasingly recognized, and the trend is that any schema should be built using a compositional approach by reusing other relevant existing specifications (if any) and by adding the specificities, in terms of concepts and vocabularies of the application domain. Under this perspective what is standardised is the mechanism for integrating multiple metadata schemas. RDF provides just this mechanism. Making a metadata standard agreed upon by a specific community interoperable increases the communication of it with other schemas, and increases the possibility so that it will be machine-processed by different systems and applications. By adopting the RDF model interoperability can be achieved not only at the schema level, but also at the record level. In fact, it becomes possible to understand that a concept is common across different repositories, and accordingly link the information. Of course, other methods to achieve interoperability (e.g., crosswalks) are not excluded.   A good practise to increase interoperability is to re-use existing properties (vocabularies) from other recognized standards, and for the new proposed ones to formally define similarities/relationships with existing ones and link them with properties, such as owl:sameAs.   Another type of relation between new defined properties and existing ones that is a good practice is to define the former to be a sub-property of the latter. A similar approach has been adopted in most Dublin Core based standards.
 
The sustainability and uptake of a metadata scheme over a period of time is an indicator of its success. Most of the times reuse and adoption of a whole metadata scheme by another community is  not feasible. This is mainly because of the differences between the specific needs of each community. The possibility is higher for parts of the metadata scheme, though. In our case, for example the reuse of mEducator controlled vocabularies about Educational Outcomes or Resource Type is feasible because the aforementioned components constitute on their own an independent piece of work that could fit in other medical communities.  One other form of reusability would be the use of mEducator schema properties or classes in the development of other schemas. A good practise to facilitate such reusability is to create schema  structures  (classes, properties, vocabularies) both independent and of wider interest/acceptance, i.e. controlled vocabularies for medical education or education as a whole, wide-acceptance properties such as "creator", "description".
The flexibility of a metadata scheme is reflected also by its suitability to be implemented in different technical environments. A good practise to develop such a flexible schema is during its design to follow more general approaches and not to be tied to specific solutions. This can be achieved by suggesting some solutions-specific properties to be optional. Furthermore, such flexibility can be accomplished by weighing the pros and cons of a specific design consideration and keeping up to date with the latest scientific trends.
 
The term readability is “the ease by which text can be read and understood” [25]. In this context, the text is the standardized format that is processed by computers but is also read by developers.  So, the proposed best practice here is that metadata fields should be logically sequenced, such that, the sequence progresses from more general information, such as title, date, rights, author, towards more mEducator-specific notions such as repurposing, companionship. In addition, the grouping of metadata fields under the same container in cases where their context is common, is a practice that enables a  human to read a metadata file. However, the depth in the structure of the metadata schema should be limited, otherwise the complexity will create problems both humans and computers. Because in the context of mEducator not only developers, but, above all end-users, who are not professional indexers, are involved in the filling of the metadata, readability, in the context of mEducater, takes on another nuance, and in this category, we can classify the user-friendly visualization of the metadata schema. Some of the most important metadata fields, such as IPR or repurposing( as described in chapter 3), were misunderstood by the users. So, a good practice would be firstly to create a human-friendly representation for the developers and then a user-friendly visualization for the end-users. The easier it is to read and understand the standardized format the less visualization changes, such as renaming of metadata labels, logical grouping of metadata fields, will be needed in order to make the metadata notions to be more immediately graspable without dedicated training.
 
 
All instantiations were technically tested against the appropriate requirements/use cases; these tests have shown that the solution, overall, meets most requirements at least to some degree. Section 4.2.2 has shown a detailed comparison of the two solutions with regards to requirements. 
Regarding the technical evaluation, best practices can be distinguished in three main topics:
  1. Centralized repository solution versus distributed and semantically linked repositories solution 1 is easier to apply, since it only requires a single code to query a unique repository of materials. Solution 2 is a more powerful tool for linking external repositories to enrich the content delivered to end users. In solution 2, each repository must be brought under mEducator schema whatever the original schema is, In solution 1, on the other hand, all content needs to be imported to a central repository. If the original content was not in mEducator schema individual content items need to be brought under mEducator schema before importing them.
  2. Solution 1 uses a mashup of code, integrates all metadata content into a single repository and communicates using JSON. Solution 2 on the other hand uses distributed repositories and semantic web services. Maintaining a single code base is easier than maintaining
    distributed services, especially if not all services are controlled by a single institution.
    Furthermore, because of the way they are built, web services are also hard to maintain because local sites do not have control of the remotely connected repository. The responsibility relies on the local host. This means that maintenance and support are provided solely by each local host and the users of the external repository rely on the local host for continuing this service.
  3. Harmonization of user interface. At present there are various mEducator tools and websites, resulting in a number of different user interfaces. While solution 1 has a single standalone interface, solution 2 has multiple instantiations. While this seems confusing to an evaluator it is worth keeping in mind that an end user would only use one of these instantiations, depending on which systems the user is familiar with, or which instantiation the users’ institution has selected. (over long and unclear sentence)  For example, Coventry University at present uses the Moodle learning management system, throughout the university and therefore a user working at Coventry University would access solution 2 via the MILES+ instantiation only.
In solution 1 data and presentation are tightly integrated while in solution 2, data and the presentation are separated, with a number of APIs handling data interaction and the different instantiations handling the presentation. In terms of scalability, maintenance and support, the recommendation depends on the institution’s capabilities and policy strategy. Solution 1 is technically easier to maintain and explain to users (teachers and students), but requires adopting a new interface. Solution 2 is more open and allows easier integration of new internal or external repositories. It is also easier to integrate solution 2 into existing systems. Hence the adaptation of solution 1 will be more straightforward for institutions not currently using any learning management systems, where as solution 2 allows integration into existing setups more easily.
 
Recommendations and best practices based on scenario based evaluation
The scenario based evaluation provided us with a broad and contextualized view on the user experience related to mEducator services. The recommendations we propose here are highly generic, and offer a straight forward checklist for the design process. The way they are formatted is, however, tightly connected to the nature of the scenarios from which the user experience has been emerging. Therefore we encourage the reader to revisit section 4.3 for a comprehensive description with details of how the mEducator services address similar issues and how the users found them to meet their goals.
1.     Aim for ease of use. Refer to the most current best practices of usability and benchmark your service with widely used tools (e.g. Google or YouTube) and their prevailing conventions of use while simultaneously acknowledging the need for functionalities validating scientific content.
2.     Aim for smooth and effective use. Make sure the technology used doesn’t interrupt or slow down the workflow or e.g. the progressive search process.
3.     Provide support for finding and retrieving content by introducing an extensive schema in a prioritized order with the most critical metadata in front.
4.     Provide support for indexing content, i.e. creating high quality metadata, by using e.g. suggestions and hints as well as dynamic, responsive annotation forms when dealing with an extensive, complex schema. Also provide workflows that anticipate the progressive learning curve related to the content novel to the user.
5.     Base the user interaction on the expected concepts familiar to the user. Simplify the novel terminology and/or utilise a vocabulary with clear descriptions of technical terms (such as IPR, identifiers, quality stamp, URI, URL, ISSN, ISBN, platform users, external users, etc.)
6.     Provide support for indexing and finding content while promoting the awareness of international standards.
7.     Support multilinguality in terms of the use experience as well as content.
8.     Be precise and highlight the advantages, i.e. the added value from using the system, e.g. offer explicit information on the type of multilingual content and system functionality available.
9.     Allow inputting metadata in various formats. Especially avoid forcing users to use e.g. licenses or identifiers that are not compliant with their content or not familiar to them.
10. Provide clear routines for validating the content available through mEducator.
11. Support community creation, i.e. sharing contacts and content with other medical professionals, students, GPs and NGOs. This is characteristic for mEducator, and it is a feature that is not offered by similar, comparable systems.
 
For reaching best practice Recommendations with respect to IPR the mEdcuator consortium clustered and collaborated closely with other similar efforts (e.g. PORSCHE in UK). So, the following set of IPR recommendation were taken from a combination of the afore-described evaluation efforts and the latter collaboration. Final recommendations may be codified as follows:
- Adoption of the Creative Commons (CC) policy
-Adhering to the requirements of each CC license type and obey its rules (i.e. enforce a non-commercial use of the resources if the non-commercial license type was indicated)
-Better explanation on the meaning of the IPR abbreviation and options
-Provision of a technical a mechanism to check the validity of the IPR
-Prevent users from linking/repurposing any resources with unclear IPRs
-Defined a policy on how to deal with contents from external repositories for which the IPR clearance is unknown
-Provide clear disclaimers. In order to safeguard mEducator against litigation for copyright or data protection (consent) violation one needs to:
- Have a policy/disclaimer
-Clearly publish policy and keep it up to date
-Train staff in the use of the policy
-Follow the policy (do what has been said it would be done)
-Some good practice disclaimers are the following: ‘this resource has been provided… use it at your own risk. If you have any concerns about material in this resource…’ or ‘that ‘this material has been produced to the highest possible ethical standards and anyone with any concerns should contact xxx in writing after which the offending material will be removed within 10 working days pending investigation’.
-Actively manage risks (see risk toolkit in www.medev.ac.uk/ourwork/oer/)
-Take out liability insurance
- Consent everything-even where ownership and patient/non-patient rights appear clear, and store consent with resource
-Update with the general rules and guidelines of the General Medical Council (or other similar bodies in EU countries). For example, ‘making and using visual and audio recordings of patients 2001’ referred to clinical care and research, but did not refer to teaching, while ‘making and using visual and audio recordings of patients 2011’ does refer to teaching.
The move towards sharing digital educational resources, for example in the UK, has accelerated institutional understanding of copyright and ethical issues (such as a need for consent for people appearing) in teaching materials. Whole programmes including curriculum documentation and specific learning resources are now routinely being shared under license. To this extent, the following points are relevant:
- Organize workshops/training in copyright, IPR, consent
-Slot in a bit of digital professionalism (literacy)
-Work with professional and statutory bodies to strengthen their guidance
-Create tools to help navigate the legal landscape
-Provide easy access to good practice
 
Finally, we would like to conform with a recent proposal on consent commons by the PROSCHE group, in order to work alongside or with creative commons as a way of demonstrating due diligence in dealing with issues of consent and using patient data sensitively in learning and teaching with specific reference to being able to share (see for example www.jiscdigitalmedia.ac.uk/clinical-recordings). Consent Commons ameliorates uncertainty about the status of educational resources depicting people, and protects institutions from legal risk by developing robust and sophisticated policies and promoting best practice in managing information [26].
 
Both solutions have pros and cons when it comes to institutional adoptability and which solution is preferred from an institutional adoptability point of view will depend heavily on the individual institution. Three main aspects to consider are the interface, the use of (meta)data and sharing (meta)data with others.
With regards to the interface, adopting solution 1 is technically easier, as it requires the installation of one interface and one repository only. If there is content already available this will have to be imported into the mEducator schema, for both solutions. However, if the institute has a policy in place to use a particular system (such as DSpace), adopting solution 2 is easier, since modules will either exist, or it is possible to develop the appropriate ones without the need to change the institutional policy.
When it comes to the use of (meta)data, it is worth noting that in the case of solution 1 the institute controls all data, whereas solution 2 relies on external repositories. Therefore solution 2, relies on the external repository being maintained by its owner and the reliability and appropriateness of the data also rely on the owner. It is likely that institutes will want to restrict which external repositories can be used, to cover themselves against possible liabilities, and to ensure reliability of the service. It is possible some institutes may require formal agreements. Both solutions require adoption of the mEducator schema. Any data brought into the solutions must be brought under the schema, and users may need some form of training in using the schema.  
When considering sharing of mEducator compliant metadata we can foresee two scenarios. In both cases permission may be needed for data  to be shared and it is possible that not all data can be shared.
  1. (meta)data is shared through institutional repositories (such as d-space in Helsinki). In addition to requiring approval, technical resources are required to make sure the repository is a reliable resource for outside users. Either the mEducator compliant metadata is shared using a custom solution developed on top of the institutions own exiting systems, which is relatively expensive, or an "already modified" system that has a plug-in is adopted (or solution 1 can be used and its repository made available), which would be "low cost".
  2. Alternatively an individual teacher can independently join the network and use one of the "centralized" mEducator portals. This raises the question whether the institute allows individual teachers to share teaching materials to other external institutes and how it views an outside system being deployed in its delivery.
  3. In due of the above, the mEducator consortium has worked on a simplified and versatile model of adopting some of the mEducator3.0 instantiations (MELINA+, MILES+ and LL+ in specific, as explained in D8.1 and will be presented in D8.2; see [27]).
 
The following Table attempts to provide a summarised overview of the user testing contexts.
Table 15. A summary of user testing contexts
 
Evaluation Place, Country, Date,
Group size
Type/Context of users
Nationality of Users
Vienna, Austria, 31st August 2011
20
Medical Education Professionals participating in the AMEE conference
From all over the world (English, Dutch, Italian, Finish, Swedish, Indian, American, Brazilian, Mexican, South African, Canadian
Plovdiv, Bulgaria, 6th October 2011
25
Medical Professionals, and Medical students (undergraduate)
Bulgarian
Plovdiv, Bulgaria, 22nd October 2011
15
Students of Dentistry & Medicine
Bulgarian
Bucharest, Romania, 21st  October 2011
15
Medical students (undergraduate)
Romanian
Bucharest, Romania, 21st  October 2011
5
Medical Professionals
Romanian
Bucharest, Romania, 21st  October 2011
15
General Practitioners
Romanian
Nicosia, Cyprus, Nov.11 2011
35
Postgraduate students on an e-health MSc course
Cypriot
Thessaloniki, Greece, 1.12.2011
130
undergraduate medical students attending the Medical Education module
Greek, African, Syrian and Palestine
Thessaloniki, Greece, 7.12.2011
10
post-graduate medical students & health professionals attending the MSc module on Digital Media in Medical Education and Practice
Greek
Various Places
from 2.12 – 16.12. 2011
18
postgraduate medical, academic medical staff & health professionals (through partners ECCA, SGUL and MUPlovdiv, AUTH)
English, Belgian, Greek & Bulgarian
Alexandroupolis, Greece, 19.12.2011
31
Undergraduate medical students attending the Molecular Biology programme
Greek
Thessaloniki, Greece, 12.1.2012
130
(in self-learning mode)
undergraduate medical students attending the Medical Education module
Greek, African, Syrian and Palestine
Vilnius, 1.2.2012
10
Health Professionals
Lithuanian
Thessaloniki, 4-5.4.2012
50
Medical and Dental Academic Staff, Health Professionals, Post-graduate Students attending the mEducator Spring School
All over EU
Thessaloniki, May 2012
160
Under-graduate Dental Students attending the module of Dental Informatics
Greek and African
 
It is worth to note, that all the above user testing sessions were conducted with the help of specific scenarios (templates are included in the Appendices) and were referring/focusing on different mEducator solutions/platforms. The mEducator Spring School in specific (one before the final row in the above table), however, has also tested different applications developed in mEducator, like those of the Problem Based Learning Approach through the use of the LinkedLabyrinth system, or the WebTrace system, or the Games repurposing approach, through the use of the mEditor (game scenario editor), or the running of the semantic medical game (Dr Hoo).
It is important to focus a little bit more on the above points, as these provides clear routes to modern resources like games, semantically rich content, medical videos, anatomical cases and simulations, but also a link with contemporary pedagogical approaches like that of Problem, Case or Scenario Based Learning (PBL/CBL).
In fact, to satisfy the above points, the consortium devoted developments (in WP5 specifically). That is, the mEducator3.0 instantiation LinkedLabyrinth+ which is merely for Virtual Patients. The latter is by definition the most contemporary problem (or scenario) based learning approach in medical education. There exist a good deal of material already in that platform (as well as in other platforms) which is entirely PBL. In essence, LinkedLabyrinth+ has been mainly developed to cover for that gap.
Moreover, during one of the mEducator Spring School sessions (http://www.mei2012.org/content/creating-searching-repurposing-medical-co...), participants were asked to follow a scenario in which more modern content like medical videos were utilised (a special instantiation, i.e. MELINA+Videos) was developed with some 4000 non traditional medical education resources, semi-automatically retrieved from YouTube. So, in the following figure, 2 (Linked Labyrinth+ and MELINA+Videos) out of the 4 interconnected systems utilise mainly non-traditional resources.
 
 
Figure 57: mEducator3.0 interconnected systems used in the mEducator Spring School with emphasis on non-traditional resources.
 
In addition, the consortium has employed a number of MMs and resources (in WP3 and WP4-6) to develop an editor (mEditor) that can repurpose game scenarios, in order to allow for the production of (additional) alternative educational content types. These have been already tried out and evaluated with real users during the mEducator Spring School (see for example session  http://www.mei2012.org/content/repurposing-scenarios-serious-games-hands for details).
 
From Table 15 above it is obvious that user testing has already taken place with different nationalities (languages) of users. Special care has already been taken during the project to allow the design of the mEducator resource description in different languages. A special section on how the mEducator schema currently takes care of that will appear in D6.2. The utilisation of SKOS (SIMPLE KNOWLEDGE ORGANIZATIONS SYSTEM) to describe the taxonomies specified in mEducator, forms also a potential asset of the mEducator approach towards multi-linguality.
In addition to that, however, special care has been taken in mEducator2.0, so that full multi-linguality is supported in all menus and metadata. Automatic translation tools like Bing from Miscosoft have also been employed to provide free translation services for online automatic translation of text and web pages, translating between many different languages. However, a different approach might also be needed, as these tools are not too effective when semantics play a crucial role, as it might be the case in Medical Education. A semantic translation approach might therefore be needed. To this extent, m
Educator has been clustering with the MORMED EC funded project (Multi-lingual & Organic Information Management in the Medical Domain; http://www.mormed.eu/project), which aims to establish and offer a multilingual thematic community platform for rare diseases as a service to groups of interested users. The MORMED team has participated in the mEducator Spring School (see session http://www.mei2012.org/content/facilitating-sharing-multilingual-content-example-mormed-%E2%80%93-demo-and-hands) and a first set of ideas for combining the mEducator and MORMED works has already been initiated.
 
It is believed that by utilizing the mEducator technologies and by using all the possibilities of the shared content a “shift towards better achievement of learning outcomes” may be achieved. This has actually been the case in an undergraduate module, that of Medical Education, within the undergraduate medical curriculum at AUTH.
The traditional weekly schedule of 2 hour lectures has been enriched with a 2 hour laboratory session in which students had a hands-on session in mEducator3.0/MELINA+ in order to search, retrieve and repurpose educational material. The course assessment was also modified through this change, and incorporated a portofolio that students maintained online in MELINA+. The whole experience seemed to be adding value to the learning outcomes of the course as it puts in practice something that student were becoming familiar in a passive way and only through lecturing. This is fact shift the focus in active learning aspects as students are asked to do things and learn through that experience.
Qualitative results of the semester evaluations seem to be promising and could argue for the success of this approach. In addition, some students were specifically asked to provide detailed oral comments on the overall paradigm shift approach as they had experienced it in mEducator during the academic year 2011-2012. The reader is referred to see the registered witnessing of a medical student in the mEducator promotional video: http://www.youtube.com/watch?v=HK5psY48kaQ  (time: 00:01:40).
 
We would like to conclude with a Technology Enhanced Learning Scenario (TEL) for the mEducator case. This scenario might illustrate how mEducator technologies can be used in different steps of the process of planning and implementing a TEL innovation in the context of a medical course/module, and a teleradiographic anatomy course in specific. The scenario is developed from the perspectives of both the teacher and a student. The core parts of the scenario have already been tested in a hands-on session offered at the mEducator Spring School, and were evaluated both by the school participants and the rapporteurs that were attending the session (http://www.mei2012.org/content/webtrace-making-anatomy-tracings-over-web...). A fully detailed version of the scenario is given in Appendix VIII.
It is believed that the above set of tests and evaluations presented in previous sections of this deliverables, together with the multilingual, modernized approaches and TEL activities will further enhance the best practices elicited by this project.
 

[1] These recommendations were further benefited from the recent Ask the Expert session on “Open Educational Repositories, Intellectual Property Rights, Creative Commons and Non-Commercial Web Licenses - Consent Commons” held within the 1st International Spring School on “Medical Education Content Sharing Technologies” which took place from Wednesday 4th  to Thursday 5th of April 2012, in Thessaloniki, Greece. as part of the mEducator activities (see http://www.mei2012.org/spring-school-programme/thursday).