Music Notation, Technical Requirements

MPEG AHG on Music Notation Requirements

http://www.dsi.unifi.it/~nesi/mpeg/ahg-mn-65-66.html

In collaboration with the MUSICNETWORK

http://www.interactivemusicnetwork.org

 

Version: 1.5

Date: 03-12-02

 

Effort coordinated by the MUSICNETWORK:

Paolo Nesi, Piefrancesco Bellini, Giorgio Zoia, Jerome Barthelemy

For a partial but extended list of contributors see at the end of the document.

 

1         Introduction

This document reports the detailed requirements of the Music Notation Representation into MPEG. This document has been created by the initial effort of the people of the MUSICNETWORK that participated at the first Workshop on Music Notation (updated name: Workshop on Music Representation) in the September 2003, Leeds, UK, organized by the MUSICNETWORK.

It should be understood that "music notation" in this document means all kinds of symbolic music notation including different styles of Gregorian Chant, Renaissance, Classic, Romantic and 20th Century styles, simplified notations for children, Braille and other forms yet to be invented.

Thus the title should be Music Representation. On the other hand, it has to be Music Notation to keep it aligned with the terminology introduced in MPEG and by its AHG. The importance of having a music representation inside MPEG should not be underestimated.

 

A separate document concerning more in details the integration of Music Notation in MPEG and the Music Notation functionality has also been prepared by the AHG on Music Notation Requirements and it has been submitted to this same meeting as M10355.

 

The requirements reported in this document should be taken into account for producing a call for technology for including a Music Notation Model into MPEG; in addition, they could be used for determining the criteria for assessing the proposals, to verify that the potential standard meets the specified requirements, and to specify the conformance point for the standard.

 

This document has two appendixes:

n      A) An excel file containing a large list of music notation symbols, their common names and classification according to the requirements reported in this document.  

n      B) A review of music notation model published on a book: P. Bellini, P. Nesi, "Modeling Music Notation in The Internet Multimedia Age", in Visual Perception of Music Notation: On-Line and Off-Line Recognition, edited by Dr. Susan E. George, press 2003.

 

Several products are currently available in the market that present some form of integration of music notation and multimedia (see the examples reported in the MPEG document M10068 Regarding the Application Requirements also available in other forms on the www site of the AHG, http://www.dsi.unifi.it/~nesi/mpeg/ApplicationRequirements-MusicNotation-1-0.html).

These products are in the areas of

  • music education (notation tools integrating multimedia),
  • music management in libraries (music tools integrating multimedia for navigation and for synchronization),
  • entertainment (mainly synchronization between sound, text and symbolic information),
  • piano keyboards with music notation and audiovisual capabilities,
  • mobile applications,
  • i-TV distribution
  • etc.

The integration of music notation in MPEG could completely satisfy the requirements of these tools and much more permitting to these tools to integrate the powerful model of MPEG for multimedia modeling, representation, coding and playback. The integration of music notation with MPEG will open the way for a large number of new applications and markets related to the above applications.

This initiative may increase the present market for music notation that presently is mainly limited to sheet music production, and may open the path to create very interesting new applications, renovating the present applications that already use some integration between multimedia and music notation.

 

The repercussions would have a wide effect on music education and on the development of music as a whole. This, in turn, would be of great long term benefit to the music industry, quite apart from the applications, mentioned in this document, which can be written and sold in the shorter term.

 

1.1      Integration in MPEG

The benefits of solving the problems highlighted above are remarkable. A unified model for music notation integrated with an MPEG standard could provide the solutions for several classes of applications including:

·        distribution of music content and courseware on media devices, i-TV and other devices such as PDAs,..

·        multimedia music distribution on multimedia devices, mainly visual and audio support;

·        fruition of composite music-related information, mainly audio and simple visual representation;

·        creating innovative applications incorporating multimedia integration with:

·        music editing and formatting (an example might be WEDELMUSIC editor with MILLA engine)

·        cooperative work on music score (an example might be MOODS)

·        producing opera content with VR-3D, synchronization of music notation, lyric and virtual modeling (an example can be OPENDRAMA)

·        synchronization of symbolic music score and real audio (an example might be WEDELMUSIC)

·        3D hand representation and synchronous score visualization (an example might be MUSICALIS, EMEDIA, PLAYPRO, GUITAR Magic)

·        synchronization of images of music score and real audio (an example might be WEDELMUSIC)

·        creating content and courseware for music education

·        examples of multimedia music content can be MUSICALIS, SMARTSCORE, WEDELMUSIC

·        cases in which the pupil can do

·        play training (examples can be MUSICALIS, PIANO TUTOR),

·        compositional training,

·        play along (see Master Play along, MUSICALIS, PIANO TUTOR, VOYETRA),

·        sing training,

·        conduction training (see example of Vienna Philharmonic),

·        improvisation training, etc.

·        production of music notation and synchronization with other media

·        optical music recognition applications

·        music transcoding: from audio to music notation via audio processing

·        establishing relationships among images of music sheets and symbolic representation

·        creating and managing content in music archives

·        examples can be WEDELMUSIC tools

·        creating the support for the interoperability among the above mentioned different applications.

·        creating musical information for consumers with different accessibility needs.

 

The missing parts to cope with these applications and requirements are mainly:

·        Music notation modeling

·        Music notation decoder

 

 

Applications and companies that may provide these technologies are listed below and many others are reported in the www page of the AHG http://www.dsi.unifi.it/~nesi/mpeg/ahg-mn-65-66.html :

·        ARCA, Italy

·        Berlioz,  http://www.berlioz.tm.FR  

·        Capella, Music Editor, Educational Tools, audio processing tools, etc., Germany http://www.whc.de

·        Encore music editor and company

·        Finale 2003, CODA, www.codamusic.com , CODA music, US

·        Graphire Music press,  http://www.graphire.com   

·        IGOR, NOTEHEADS,            http://www.noteheads.com

·        Lime music editor and company

·        Mozart, Mozart, http://www.mozart.co.uk

·        MusEdit, http://www.musedit.com  

·        MUSICALIS; HTTP://www.musicalis.com   

·        MusicEase, it includes an automatic accompaniment tool.

·        Nightingale (Mac), note-processor, from Adept Music Notation Solutions (USA)

·        Recordare with MusicXML, for the Music Notation format

·        SCORE: http://ace.acadiau.ca/score/LINKS3.HTML , music editor

·        Sibelius Software, UK, www.sibelius.com

·        VIVALDI, Amadeus, http://www.vivaldistudio.com

·        WEDELMUSIC, WEDELMUSIC Authoring, http://www.wedelmusic.org, Italy

 

2         Domains of Music Notation

The modeling of music notation is a rather complex and composite problem. Music notation is a multi-layered piece of information, which may be used for a wide range of different purposes: from audio coding and entertainment to music sheet production, music teaching, music analysis, content query, provision of enhanced or adapted music for consumers with specific needs, etc. In the current Multimedia and Communication age many new music-related applications are strongly attracting the market attention and most of them will become more and more widespread in a short time. In order to identify a unique notation format to satisfy all these application fields and to set the basis for the new forthcoming applications, several aspects have to be considered, ranging from information modeling and format to integration and synchronization of the music notation information into other media and cross-media tools and formats.

The evolution of information technology has recently produced changes in the usage of music notation, transforming the visual language of music from a simple written “graphic alphabet” for creating music sheets to a tool for modeling music in computer programs, cooperative work on music and other multimedia integration tasks, associating actions with music notation features during playing, etc. More recently, millions of music users have discovered the multimedia experience, and thus, the traditional music notation model thus needs to be replaced with something much more suitable for multimedia representation of music. The improved capabilities of computer programs are going to solve many of the early music notation formatting problems encountered by the adoption of powerful tools based on artificial intelligence technologies.

 

The present models for music notation are unsuitable to cope with many innovative applications. A deep analysis suggested that the music notation mainly comprises of four domains of representation. Elsewhere, these domains may be referred by different terms, but here we categorize the domains thus:

·        Audio: (also called gestural, or performance) description of how a given note or chord has to be played in terms of parameters in the frequency domain, orchestration, sampling of audio signals to be used as wave tables to play notes, etc

·        Logical: description of the music notation structure in terms of sequence of notes, rests, chords, etc. organized into instruments, voices, measures, etc. This also includes the expression symbols that represents how a certain note or sequence should to be executed: markers (general terms used in some programs to identify accents, ornaments, etc.), accents, ornaments, accidentals, key signatures, clefs, dynamics, key signatures, crescendo and decrescendo, phrasings, bowings, instrument indications, metronomic indications, multilingual lyric, etc. Many performance parameters (e.g., MIDI) can also be included in this domain. Also needed is the modeling of rules to generate alternative music notation for specialized consumer groups to be associated with the logic aspects. In this case, strong relationship with Audio Domain is needed. All aspects that are common to Audio and Visual aspects should be placed in the logical domain to create what we calling the bone of the music representation.

·        Visual: parameters describing, the visual representation of the logical content as absolute and/or relative positions (spacing), colors, stem direction, beaming structure and slope, refrain/chorus, generic rests, slurs shape (knots based or other), 8va or 8ba or 15ma, repetition symbols, barlines, etc., and also the eventual image representation of the score, graphic representation of modern music notation symbols, formatting representation of the main score and individual parts in terms of justification parameters and page layout, ordering for symbols positioning around the figures, etc. (for example, which symbol has to be positioned closer to the notehead if a staccato, a slur, and a fingering are present on the same note) This domain implicitly also includes the formalization of the relationships among music notation symbols. It also includes the description of the image sheet and its relationships with the music notation structures and elements (parts, measures, etc.)

·        Graphic representation deals with the mechanisms to produce the music notation on an image sheet either as print output or screen canvas. This is performed by using graphic primitives and fonts, and associated parameters such as size, font and style together with mechanisms to avoid confusions (e.g., collisions, overlapping, etc) among graphic details. Details from the visual domain have to be considered as high level aspects/directives for the graphic representation. Separating graphic and visual aspects gives the possibility of generating alternative visualizations of music notations, for example for visually or print impaired users.

·        Analytical information: information about the work, which might include bibliographic information, as well as interpretive information that ranges from phrase markers and roman numeral analysis of underlying harmony to a Schenkerian graph

 

The following figure reports some of the domains and the relationships with MPEG solutions. This has been discussed at the first workshop on Music Representation and more in deep in a document visible on:

 


The requirements reported in this document are mainly related to the Visual and Logical parts.

3         Requirements of music notation model in MPEG

The following list of requirements is referred to the Music Notation requirements for including music representation inside the MPEG. These aspects are mainly focused on:

  • Defining the most relevant requirements for modeling Logical and Visual aspects described above,
  • Graphical aspects will be supported by BIFS and FONTS
  • Audio aspects will be integrated with those supported in MPEG in SASL and SAOL.

 

The requirements are divided in:

  • General requirements,
  • Logical requirements
  • Visual Requirements
  • Extended Functionalities requirements
  • Integration Requirements

3.1      General Requirements

·        scalability: it has to be possible to write simple music notation pieces files without spending time and token to describe all the music notation context (for instance, it has to be possible to write simple music pieces describing only the needed parts avoiding to spend time and language tokens in describing the full context, that could be: collection, sections, parts, voices, etc.). At the same time, the model has to support the writing of complex structure of music pieces with multivoices, multistaff, etc., according to the applications identified. This will make it suitable for simple devices and applications such as those supposed for mobile equipment. This requirement implies that the music notation decoder has to consider for each aspect that can be missing a default value and behavior.

·        interoperability: interoperability is required among different devices: i-TV, Mobile Devices, PDAs, Tablet PCs, electronic musical instruments, special hardware, cars, etc.; and also among software applications: music editors, music tools for educational purpose, music viewers on the i-TV or PDAs.

·        extensibility: includes the possibility to create new symbols with their related rules or code for formatting (conversion to the visual domain and from this in graphical domain) and executing/playing (conversion to the audio domain). Please see the requirements of formatting.

·        support for domains: the language has to support a sharp distinction between music notation domains. In the language has to be clear which aspects are logical, visual and graphics and possibly audio and analytical.

·        decodable: the model has to permit the decode of music notation with different decoders to obtain different views of the same music representation (main score, parts, Braille, Gregorian, spoken, etc.). This can be done by using additional information or automatic formatting engines. This is very useful to work on different devices (piano keyboards, PDAs, mobiles, i-tv, sheet, screen, etc.), etc. The music notation integrated with audiovisual could be streamed with video, audio, .. on mobiles, etc. (see MPEG4 model). Different decoders can be realized according to different models of music reproduction and visualization. This requirement has been detailed in the document M10355.

·        distribution and Protection: the music notation model has to support the distribution on Internet integrated with any kind of audiovisual content to satisfy the needs of realizing the above mentioned applications. To this end, the several activities that can be performed on the music notation model have to be classified to identify the Digital Rights Management rules, for example by defining specific rule with the MPEG21 to control all the possible functionalities of a music notation content:

o       playing, printing, annotation, execution, formatting, transposing, etc.

·        editable: The music notation has to be editable with some music editor. The format has to be supported by some specific editors for changing in deep all the details of the symbols involved.

·        readable: The music notation model should be readable by humans when will be in a non binary form. This probably means that we need to have some XML model.

3.2      Extended Functionalities requirements

·        An Univocal reference ID for each music notation symbol in the music piece. This is useful for Multilingual and Annotation addition.

o       Multilingual Lyrics: Each symbol or at least the most relevant should be univocally identified for associating to them the lyric syllables. The model in this way may include the possibility of loading, on the same music representation, lyric texts of different languages. This is a very important feature for realizing multilingual CD-DVDs without constraining the producer to create different versions of the same music notation model. This requirement is also useful for the production of multilingual Karaoke.

o       Annotation: the model has to permit the insertion and the creation of annotations by the final users. The annotation may be musical entities, text, and HTTP documents or links. They can be useful for adding personal comments to a digital object, to find in a second phase the specific point of the entertainment content, etc. This means that each annotation has to be associated with each specific element of music notation univocally, symbol by symbols. The list of annotations may be navigated and listed. Annotation coming from several users may be merged and distributed, saved and loaded. Examples of annotations are:

§         Reference Link, as described in the following

§         Traditional music notation elements and symbols: accidentals, fingering, bowing, adding of attention marks, adding of caution accidentals, dynamics, tempo marking, breath marks for wind instruments, special marks such as pedals for piano, harp, tablature for guitar, cue notes…

§         hand-produced annotation (lines, frames, specific shapes…),

§         multimedia content as audiovisual (audio, video, still images, animations, actions to the external world and on physical devices).

·        Reference links for each music notation symbol. These are useful for realizing:

o       User Navigation (relationships or links): the model has to permit the insertion and the creation of relationships between each notation symbol (or on at least the most important, such as the Music Notation Events and the structural aspects) and audiovisual elements (text, documents, audio pieces, video, music pieces, http links, etc.). The relationships-links should be established during the production of the content and can be navigated by the final user. They can be regarded as relationships with internal content elements or external links. They can be useful for:

§         explaining the music notation,

§         navigating from the music notation elements to other audiovisual aspects in a MPEG4 file,

§         connect the content with web resources.

o       Automatic Execution: the music notation can be executed without playing/rendering it for the simple purpose of activating specific actions associated with execution of a specific music notation symbol. This can be useful for activating the execution of some action when a music note is executed (this can be realized by including in the model some specific annotations) for:

§         educational purpose

§         creating applications in the field of art and theatrical activities.

§         playing some audiovisual or

§         activation of some external events.

·        Formatting: the decoder of music representation has to properly manage all the information for reproducing the music notation on different formats and page size (computer windows, music sheets, etc.), different type of decoders may be considered (e.g., main score, parts; Braille or CMN, etc.). The information needed to reproduce the music on different formats have to be included into the music notation model, the rest has to be part of the context.

·        Query by content: the model has to support the query by content by integrating audiovisual and music notation, usage of music notation model for verifying the match with MPEG7 descriptors of music that presently are very simple, increasing the expressivity of MPEG7 music descriptors.

3.3      Logical requirements

The detailed requirements of a music notation model are not needed for creating a call for technology and for enforcing the music notation into MPEG. On this basis, a call for technology could be prepared. The present requirements regarding the logical aspects of music notation are those which have been identified to be strictly necessary to:

·        guarantee the above mentioned general requirements.

·        have a music notation format correctly working in the MPEG environment

·        guarantee the exploitation of the functionalities which are needed to support the innovative applications listed in the WWW site of the AHG.

 

The scope of this model has been preliminary limited to include:

•          Musical structure representation

–         Structuring the logical representation

–         Including visual aspects as separate and well identifiable aspects

•          Western music notation

•          Tablatures and Percussions

Recently, the logical part that we are going to define is absolutely broader.

 

The most relevant Requirements of the Logical aspects of Music Notation are:

·        Structure: the structure of music notation model has to model the following aspects

o        Collection: a set of operas or collections plus related metadata.

o        Opera: a set of movements plus specific metadata.

o        Movement: a non interrupted (of musical time) single division of music, which is a collection of MVP, from 1 to N; plus specific metadata.

§         Visual: main score, size of the page sheet and printing parameters, computer window visualization parameters.

o        MVP, Multi Voice part: a part with one or more voices, a set of voices that are synchronized, they can be arranged in different manner on staffs/staves, plus specific metadata.

§         Logical: executable or not.

§         Visual: hidden or not, number of staves and their scale/size, size of the page sheet and printing parameters, computer window visualization parameters.

§         Graphic: size of the lines, etc.

o        Voice: a sequence of Music Notation Events, MNE, which can navigate on different staves, etc.

§         Logical: executable or not, type of voice (pentagram, tablature (number of strings), percussions (number of lines), etc.)

§         Visual: hidden or not

o        Ossia: an alternative or an explanation of music representation symbols or set of them. It is an alternative MVP associated with the referred MVP by using a starting and an ending MNE.

§         Logical: executable or not, executable in the place of referred music section.

§         Visual: hidden or not

o        Music Notation Event, MNE: An MNE is something that occurs along the MVP. Each event has:

§         a starting instant that can be started in conjunction of the end of one the previous MNEs. Thus the events can be overlapped.

§         a duration expressed in time units. The time units are capable of describing all note duration plus augmentation dots and touples duration. The visual features of the MNE include also the visual duration, etc. The duration can be 0.

o        Segment: portion of a Movement identified with a starting and ending Music Notation Event.

o        Refrain: manage the modeling of compressing music model that presents segments that have to be executed more that one time or in a different order in which they have been initially encoded. In the Music Notation model in MPEG is convenient to have the music linearly exploded and not compressed to avoid the multiplication of references. Thus a refrain results to be mainly a visual aspect.

§         Visual: A Refrain is represented by specific visual aspect on barlines, with labels and/or jump symbols associated with barlines.

·        Music Notation Events: Some examples are reported in the following to give more evidence at the needed music notation requirements

o        Barline: a barline can be regarded as the marking of a measure and it is regarded as a special event. This solution simplifies the problems of music analysis, execution, and justification of main scores that have parts with different time signatures (such as the Don Giovanni piece). A barline is a MNE with duration equal to the sum of the durations of the MNEs that are present until the next barline. The Barlines have a unique ID as the other music notation symbols. Each barline as a:

§         Visual:

·        there are several types of barlines: simple, double, hidden, dashed, etc. Some of them are associated with the refrains, and thus their visual representation has a double meaning, as segment end, as delimitation of measure, as end-start of refrain.

·        End line barline or not

·        End page barline or not

·        Justification parameters for:

o       Formatting the measure in several formats such as: main score on resize-able computer windows, single part on resize-able computer windows, main score on a music sheet single part on a music sheet. The justification include a type (linear, log, etc., and a scale factor) for each of the formats identified.

o       Context and the Transposed Context with a duration equal to the next change of context   The Context is defined as:

§         Clef or clef change, they are typically smaller in size that normal size. They are typically placed before the MNE from which the clef has to be intended changed.

·        Visual: position

·        Graphic: font

§         Key signatures changes, they are typically accompanied by accidentals adjusting the tonality from the previous to the next. They are typically placed before the MNE from which the key signature has to be intended changed.

§         Time signature changes, they are typically placed before the MNE from which the time signature has to be intended changed.

·        Visual: position

·        Graphic: font

o        Notes, with all their features and markers that are discussed in the sequel:

§         Logical: pitch, cue, accidental, Marker (see in the following), active (to be played or not), logical duration (user for the execution), reference to the audio section of MPEG.

§         Visual: visual duration, number of dots, note head, size (small or normal), colour, direction of the stem, visible or hidden, etc. On the basis of the visual duration and on the basis of the presence of dots the correct visualisation is produced.

§         When logical and visual durations are missing the note/rest is ghosted, when logical duration is present and visual is not present it is hidden, when visual is present and logical is not present it is graced. When both are present they are used for different purpose.

§         Graphic: font

o        Rests, their features that will be discussed in a second phase are

§         Logical: active (to be played or not), logical duration (user for the execution), reference to the audio section of MPEG.

§         Visual: visual duration, number of dots, position (duration based or central), highness, size (small or normal), colour, etc. On the basis of the visual duration and on the basis of the presence of dots the correct visualisation is

§         Graphic: font

o       Chords, Vertical groups of simultaneous MNE with the features of the notes. Some of the features are associated with the chord other to each single note. Those associated with the whole chord are:

§         Logical: pitch, cue, accidental, Marker (see in the following), active (to be played or not), logical duration (user for the execution), reference to the audio section of MPEG.

§         Visual: visual duration, number of dots, direction of the stem, etc. the chord can be represented in several manners, by using a freatboard or by means of textual annotation. Each note of the chord may have different colour, and specific accidentals and marker such as fingering, etc. On the basis of the visual duration and on the basis of the presence of dots the correct visualisation is

§         Graphic: Each note of the chord may have different font

o        0 duration MNEs such as:

§         Anchorages: hidden points for attaching: crescendo, diminuendo, external events of synchronisation, lines, markers not associated to notes and rests but that have a temporal position, etc. The anchorage temporal position is given in percentage with respect to the duration of the previous MNE in the MVP. For example, thet may have associated with breath and/or attention markers.

·        Horizontal Groups, HG: are structural elements that present a starting and an ending MNE. Each of them has some additional features. For example:

o        Beams: to see beams as HG permits to realize beams crossing barlines

§         Visual: the slope of the beam, the thickness of the beam, the presence or not of flags in some cases, the direction of the beam, the position of the beam in the middle or out of the staves in a multistaff part. etc.

o        Tuplets: they can be nested and, they have a number associated with. On the basis of the number and of the duration of the MNEs included the Logical Duration of these MNEs is changed.,

§         Visual: the number can be written or not. can be drown in different manners (squared and rounded), up or down.

o        Tie: connection between to successive notes or chords of the same pitch, it has a double meaning as visual and structural description of the harmony

§         Visual: shape of the tie, up or down, etc.

o        Slurs:

§         Visual: up and down, shape of the slur, 2 or more knot points for creating different type of slurs.

o        Bend: also called brackets,

§         Visual: they can be continuous, dotted, dashed, up or down, etc.

o        Lines:

§         Visual: they can be continuous, dotted, dashed, up or down, etc.

o        Double event Piano pedals:

§         Visual: up or down, etc.

o        Dynamics: crescendo, decrescendo, etc.

§         Visual: they can be continuous, dotted, dashed, etc.; up or down, etc.

o       Double Events Ornament: glissando, wave, etc.

§         Visual:  up or down, etc.

o       Octave and 15esime: change of octave, typically only visual aspect.

§         Visual: up or down, etc.

·        Markers: are music notation symbols associated with MNE, they may have a logical meaning that may impact on the music execution and a visual representation. The most relevant categories of Markers are reported in the following. For each of them:

o       Logical: some of them may have a specific music notation meaning expressed in terms of music notation to be used during the execution for better rendering of music influencing the music note playing. In this case, a semantics has to be associated with in terms of a music Segment.

o       Graphic: fonts, shape in terms of graphic primitives in SVG.

o       Visual:

§         Default position given up or down with respect to the note and its stem when applicable

§         Specific displacement with respect to the default position given in music units

            For example:

o        Accents-expressions: staccato, tenuto, portato, etc.

o        Generic User expressions: added by the user or for custom markers

o        Simple event Ornaments: turn, double turn, arpeggio, arpeggio up, arpeggio down, turn back, turn up, etc. They may have one or two accidentals.

o        Instrumental indication: bow up, bow down, etc.

o        Mutes: with the E, with +-, with “con sordina”, “senza sordina”, etc.

o        Refrain Marks: they can be placed only on barlines

o        Tempo marking: it can be associated with any MNE. It includes a reference duration note, a number and a text.

o        Dynamic Text: f, p, ff, fz, pp, ppp, etc.

o        Text: any textual indication

o        Harmonics: round and diamant

o        Fingers: to state the finger that has to be used and how

o        Single event Pedal and holes: Arpa pedal, clarinet holes, recorder holes, etc.

o        Percussion markers: stick, snap pizzicato, maracas, soft, wire, damp, etc.

o        Strings: specification of the string to be played.

o        Change of tonally: for percussions, etc.

o        Spaces: vertical and horizontal spaces. Vertical spaces are sized in terms of music unit. Horizontal spaces are sized in terms of percentage of duration of the MNE at which they are associated with. They are an only Visual aspect.

o        Etc.

·        Lyric: lyric test is comprised of Syllables and has to be multilingual. Each syllable can be associated with a MNE and in some cases with two MNEs (one for the start and one for the end).

o       Graphic: font

·        Observations: They are external observations of the music representation that can be used by the DECODER to present the music in different manners or to access to the music in differently structured manners.

o        Vertical Section (also called harmonic section in some cases): a set of MVP, to describe as the orchestra is decomposed, for example the group of strings in the operas. A VS may have a visual representation or not: staff brackets of different type: rounded and graph.

o        Horizontal Section: an arbitrary piece of MVP. A HS can be used for identifying segments in the music piece and not for structuring the information. It is a high level descriptor associated with the music by using a starting and ending point. 

·        Transposing: the model has to permit the production of transposed music notation content without losing the initial information and tone. This means that in each barline, it has to be stored the original Context and the Transposed Context. The transposition should be performed on all music piece of a single part or on a smaller part of it.

·        Metadata: have to include

o       classical models such as UNIMARK for classification,

o       some and multiple identification codes and:

o       incipit description,

o       tonality,

o       Ambitus,

o       Instrumentation,

o       critical edition and comments, etc.

 

General remark: each music notation symbol, MNE, observations, markers, HG and structural aspect have to provide:

·        Univocal reference ID for each music notation symbol for syllables and annotations

·        Reference links from each music symbol for user navigation and automatic execution

 

3.4      Visual Requirements

Visualization of:

·        Dynamic changes of visual aspects according to the device on which the visual representation is performed: windows, page layout, Braille page, etc.

·        Justification information for each measure and music segment, no direct graphical information have to be included into the music notation model to allow the reformatting according to the device size. Possibility to

o       Justify main score and parts, including MNEs without duration, grace notes, etc.

o       justify on the basis of the information contained in each measure (barline) and by considering the visual duration of MNEs, simultaneities, the size of the fonts, etc.

o       impose the values of justification of each measure and simultaneities, and/or

o       delegate the DECODER its definition

·        Vertical Staff Arrangement: the arrangement of the staves along the vertical axes, distance among parts or staves in general: Possibility to

o       impose the values, and/or

o       delegate the DECODER its definition

·        Line breaking: possibility to

o       impose the symbol/barline for placing the Line breaking, and/or

o       delegate the DECODER its definition

·        Page breaking: possibility to

o       impose the symbol/barline for placing the Page breaking, and/or

o       delegate the DECODER its definition

·        Position of symbols around the event based symbols (such as notes, rests, etc.): position and ordering of the symbols. Their position can be made on the basis of

o       specific measures given in musical units (the distance between two staff lines), this means that the arrangement of  the symbols is fixed and can be changed and imposed only in the phase of authoring; or can be

o       automatically produced by the DECODER by using formatting Rules and table in  for positioning symbols such as: order with respect to the note/chord, position and bounding box for each font element

·        Guitar Chords in different formats: fretboards and textual, if notes (direction of the stem, position of the notes on left and right, etc.)

·        Continuous Bass in different formats

·        Note Stem direction according to

o       Formatting Rules, or

o       As imposed by the code

·        Slur, dynamics, tie, etc. direction, according to:

o       Formatting Rules, or

o       As imposed by the code

·        Slur, dynamics, tie, glissando, shapes according to:

o       Formatting Rules, or

o       As imposed by the code

·        Beam grouping, beam position, beam direction (up, down or middle) and slope according to:

o       Formatting Rules, or

o       As imposed by the code

3.5      Integration Requirements

·        Synchronization: the model has to allow realizing synchronizations among the music notation model and other elements of the audio visual file. The synchronization has to be possible between the music notation and the audio, video and also with the music reproduction from SAOL-SASL.

o       For instance establishing that a note will have to be played when a given audio sample is played or a video segment is shown.

o       When the music playing change the execution rate the synchronization of music notation execution has to follow the changing in the rate. This is very useful for education purpose. For instance to see in details the movement of the musician hands and at the same time to read the music notation and hear the corresponding produced notes.

o       This requirement can be useful for the production of several different types of Karaoke.

o       A typical application of synchronization is the realization of a score follower that permits to show the music notation in synchronous with a played music. It can be useful for educational purpose.

·        Playing/Rendering: The music notation has to be executable in conjunction with SAOL and SASL or with MIDI. Deep relationships have to be identified and defined among the Music notation representation and the coding of audio in MEPG4. This requirement also is related with the synchronization. For example, the visualization of the music notation synchronous with the playing of the music in SAOL or MIDI.

 

4         Music Notation Decoder and Formatting capabilities

One of the main tasks of the decoder is the production of the visual aspects from the logical description of the music notation. The requirements of the visual aspects have also a strong impact on those of the decoder.

The decoder can be realized with one or more internal decoders and have to be capable to produce from Logical and Visual music notation information some output according to some parameters, for instance to produce:

·        Main score or parts, selection of a part, specification of formatting modes

·        Visualization of music on windows or page layout

·        Braille notation, on printer and screen

·        Verbal description of music notation, spoken music

·        Piano Roll production.

Other representation of music produced on the basis of the information contained in the Logical and Visual parts, and other produced by the decoder itself at run time.

We can have several kinds of decoders that may range from:

·        one which uses only the visual information provided in the music notation data (for low resources platforms) to

·        one which can provide a visual representation starting from the logical information and few visual information using some kind of formatting engine. This is necessary when the content visualization has to be adapted to a specific context.

The different decoders may have different capabilities in automatically formatting the music information, for example processing a part of all the following set of rules. The rules and the rule structure could be different depending on the type of different decoder.

 

The formatting rules should be placed on some style file, as well as the fonts have to limited on the graphical parts of the music notation model. Some examples of formatting rule include:

·        Positioning: rules for placing music notation symbols around the notes and rests.

·        Ordering or priority: rules for stating the ordering by which the markers have to be drawn around the notes. For instance, staccato, tenuto, slur, fingering, can be a good order and not the opposite.

·        Justification: inclusion of justification parameters, liner or logarithmic for each measure or music segment or interval

·        Line breaking: rules for identifying the point in which the music notation may finish the line. The end of line can be also imposed by the music notation producer.

·        Page breaking: rules for identifying the point in which the music notation may finish the page. The end of page can be also imposed by the music notation producer.

·        Compression: rules for depicting simple symbols in the place of their complex semantics in music notation. For example, rules for stating when a generic rests (rest stating that the whole measure is mute) or multirest (rest stating that there are several measures of rest) has to be written instead of placing several detailed rests to reach the corresponding value.

·        Beaming: rules for grouping notes and chords in beams on the basis of the time signature and the context. Rules for deciding the slope of the beam, rules for deciding if the beam has to be up or down, or in the middle in a multistaff part.