This document proposes
a solution for describing the midi System Exclusive implementation of any midi
capable device.
Date: 24 September
2006
Author: Marc Jacobi
(obiwanjacobi@hotmail.com)
Document status: draft,
version: 0.2
Content:
The goal of the Midi
Device Schema is to unambiguously describe the midi capabilities of a midi
enabled hardware (or virtual) device focusing on the System Exclusive
implementation of that device. This description can then be interpreted by
software and used to provide structured System Exclusive parsing and compiling
of
Over the years the
System Exclusive part of the midi protocol has been proven to be a pain.
Manufacturers implement their System Exclusive messages in what is in their
view the best possible way (best meaning cheap in most cases). Because the System
Exclusive messages has no predefined structure overall (other than start and end
markers), they need to be handled specifically and separately in the
application code of midi software. This leads to application developers having
to write specialized code for each device and not being able to support all
devices. If, as a user, your midi device was not on the supported devices list,
you could not use the software.
The Midi Device Schema
aims to change that. In order to tackle this problem, the proposed solution
uses metadata to describe the Midi System Exclusive implementation of a
specific midi device in a schema.
Although this
specification only deals with the textual Midi Device Schema declarations I
like to give you a baseline-architecture to position the explanation of schema
content in.
The following figure
displays this architecture using a receive scenario.
The Data Object Model
component is instantiated for one specific device for which to encipher the
midi data (both send and receive). The Midi Device Schema (file or otherwise)
is located and parsed into the Schema Object Model. This object model is a
read-only, in-memory representation of the Midi Device Schema. The Midi Device
Schema is expressed using the Midi Type System and optional custom types. The
Data Object Model component will create the necessary Data Converters for this
particular Midi Device Schema. Next the application code routes the incoming
Midi Data Stream through the Data Object Model which in turn calls into the
Data Converters component to convert the physical midi data into a logical
in-memory representation that is passed to an Application Component for further
processing in the application.
|
Figure 1: Basic
architecture
Sending logical midi
data from the application to the midi device would require the application to
pass this data into the Data Object Model which in turn will call into the Data
Converter component to transform the data into physical midi data according to
the Midi Device Schema.
All behavior of
transforming midi data from physical to logical and visa versa is driven by the
declarations in the Midi Device Schema.
The requirements laid
out for the Midi Device Specification are as follows:
1.x |
Global Requirements |
2.x |
Transformation
Requirements |
3.x |
Extensibility
Requirements |
These requirements
represent some of the variations encountered while doing a short analysis of
several midi Device System Exclusive implementations (Roland and Yamaha).
# |
Requirement |
Solution |
1.0 |
All oddities of
packaging 8-bit data in 7-bit Midi System Exclusive must be abstracted from
the developer. |
Logical midi data is
semantically equal to the physical midi data but presents the data in a
natural way to the application using “native” data types. |
1.2 |
Validation of data
based on the Midi Device Schema. |
The Schema itself
contains all validation rules and can be used to validate incoming logical, and
if desired, physical data streams. |
2.0 |
An addressable index
into the schema must be provided. |
- no
solution yet. |
2.0.1 |
Repeatable records
must be supported (addressable index) |
- no
solution yet. The data record (complex) type will be used as boundary for the
repeatable records. |
2.1 |
Multiple field
transformations must be supported. |
The (data) Converter |
2.1.1 |
Specify a dummy
field. Can be used to align a data record on a specific boundary. |
The midiNull data type provides a means to specify dummy
fields. |
2.1.2 |
Allow a logical
value to be a composite of more physical values. It must be possible not to
use all the bits in the physical values (for instance: 3 physical bytes
filled with (LS) nibbles make up a logical 12 bit value). |
A custom data type
can use a union to produce these results. Also refer to the (complex) data
record. |
2.1.3 |
A textual string field
value. Multiple characters represented as one logical value. |
The midiString data type forms the basis for reading textual
string data. |
2.1.4 |
Allow a Boolean field
value for any bit. |
The midiBit0 –
midiBit7 data types will allow a Boolean value for any bit. |
2.1.5 |
Constrain a field value. |
Any custom data type
can specify constraints using Xml Facets. |
|
Merged value. One
physical byte that contains multiple logical field values spanning one or
more bits each. |
A data “carry” will
provide this functionality during a To-Logical or To-Physical data transformation. |
|
Enumerations. A
sequence of named values. |
Any custom data type
can use the Xml Enumeration Facet to fulfill this requirement. Use the midi:name attribute on each
enumeration facet to name the value. |
2.2 |
Multiple data record
(=multiple fields) transformations must be supported. |
The GroupConverter |
2.2.1 |
Bit shift algorithms. |
The extensibility of
the GroupConverter provides in this need. Some (unknown
which) bit shift algorithms will be implemented. |
2.2.2 |
Checksum
calculations. |
Typical checksum
algorithms are provided as a group converter (data record base type). Custom
checksum calculations can be made by declaring a custom record type and extending
the group converter. |
3.0 |
The
physical-to-logical and visa versa transformation of data should be
extensible. |
A factory pattern is
used to create the data converters. All types from the Base Type System are
provided. Custom types can be added. |
|
|
|
Because the data
converter framework is extensible, any requirement not listed here can probably
be implemented through a custom converter or group converter.
The term Schema
represents a description of a digital data structure; midi messages in this
case. The proposed solution sets down a type system as a basis for describing
the device’s System Exclusive implementation. The schema describes all aspects
of a System Exclusive message; from message exchange patterns to field data
types.
The choice was made to
express the Midi Device Schema as Xml Schema xml. Because almost all of the
constructs needed are readily available in Xml Schema and the parsing
technology is free.
There are some liberal
interpretations of Xml Schema in the Midi Device Schema language.
Xml Schema |
|
simpleType |
A data type to be
used in a field |
complexType |
A logical field
record; grouping of fields. |
element |
A logical field
declaration |
union |
Mask, shift and bitwise-OR
a data byte |
fixed |
A fixed midi
protocol or device specific value. |
abstract |
Used on a field type
to indicate a place holder to be overridden by derived schema types. |
nillable |
Specified on a field
to make that field optional. |
facet |
A constraint on the
value of the field. |
annotation |
Used for
documentation. |
midi:name |
A custom name
attribute used to name facets (enumerations). |
The following sections
show how to declare Midi Device Schema constructs in Xml Schema.
The examples use “mt” as the namespace alias for the midi base type’s namespace and the “xs”
namespace for the Xml Schema elements.
A Midi Device Schema
declaration is exactly the same as an Xml Schema declaration. The following
example illustrates a basic schema declaration.
<?xml version="1.0" encoding="utf-8"?> <xs:schema targetNamespace="[name
of schema]" xmlns:xs="http://www.w3.org/2001/XMLSchema"> … </xs:schema> |
Note that the targetNamespace is used to identify the Midi Device Schema.
This name should be chosen such that it uniquely identifies the midi device and
its System Exclusive implementation. Schemas with the same targetNamespace
are considered to be the same schemas.
Including another Schema
Using the standard Xml
Schema includes other schemas (the base types for instance) can be included
into a custom schema.
<xs:schema targetNamespace="[name
of schema]" xmlns:xs="http://www.w3.org/2001/XMLSchema"> <xs:import namespace="[schema]"
schemaLocation="MySchema.xsd"
/> |
Declaring a data type
Data types are the
basis for converting physical midi data into logical midi data and visa versa.
The data converters are created based on these data types.
This example shows the
mother of all data types, the midiByte.
<xs:simpleType name="midiByte"> <xs:restriction base="xs:unsignedByte" /> </xs:simpleType> |
Note that typically
custom data types are derived from existing data types. See also Inheriting a data type.
Declaring a data record
A data record is a
collection (sequence) of fields that logically make up a unit. How a device’s System
Exclusive data fields are grouped into records is a choice of the schema author.
This example shows the
template for a System Exclusive message. Note the begin- and end-markers of the
midi SysEx and an abstract type as a place holder for
the message body content.
<xs:complexType name="midiSysExMsg"> <xs:sequence> <xs:element name="SOX" type="mt:midiSysEx" fixed="240" /> <xs:element name="Body" type="midiSysExMsgAbstractBody" /> <xs:element name="EOX" type="mt:midiSysEx" fixed="247" /> </xs:sequence> </xs:complexType> |
Declaring a field
A field represents a
logical value in a SysEx message and is always
declared as part of a data record.
This
example show the
declaration of the Universal SysEx Id field (using a contrained data type).
<xs:complexType name="…"> <xs:sequence> <xs:element
name="UniversalId" type="mt:midiUniversalSysExID" /> … </xs:sequence> </xs:complexType> |
Note that a field’s
type can be either a data type (representing a concrete value) or another
record type (representing a sub-record of fields).
Inheriting a data type
Currently only
restriction-inheritance is supported for data types.
The example shows the
declaration of the midiSysExData data type that is
constrained to bits 0-6 (not shown).
<xs:simpleType name="midiSysExData"> <xs:restriction base="midiByte"> … </xs:restriction> </xs:simpleType> |
Declaring custom data
types is a matter of picking the most appropriate base type and optionally
adding constraints.
[Implementation:] In order to overrule data converter functionality
one could declare a new data type and inject their custom data converter for
that data type by providing a converter factory.
Inheriting a data record
A data record (complex
type) can be inherited in order to override abstract fields in the base type.
See also Overriding a
field.
The following example
shows the declaration of a universal SysEx message
based on the midiSysExMsg template presented earlier.
<xs:complexType name="midiUniversalSysExMsg"> <xs:complexContent> <xs:extension base="midiSysExMsg"> <xs:sequence> … </xs:sequence> </xs:extension> </xs:complexContent> </xs:complexType> |
Note that deriving
data records (complex types) is by extension, not restriction as with data
types.
Overriding a field
In order to override
an existing (abstract) field on a base type the names of the field should match
exactly.
Note that only fields
whose type is another data record type can be overridden.
In the following example
the Body field of the midiSysExMsg base type is
overridden with a new type. The midiUniversalSysExBody
is a data record type that declares all the fields for a universal System
Exclusive message.
<xs:complexType name="midiUniversalSysExMsg"> <xs:complexContent> <xs:extension base="midiSysExMsg"> <xs:sequence> <xs:element
name="Body" type="midiUniversalSysExBody"
/> </xs:sequence> </xs:extension> </xs:complexContent> </xs:complexType> |
Field overrides are
name-based and the type of the field in the base does not have to be abstract
in order to be overridden. So whenever a derived data record specifies a field who’s name is present in its base record, it will override
that field with the (data) type specified. The field (data) type in its base
will not be used, unless the new field (data) type derives from it.
Note that it is
permitted to specify fields with the same name in different (complex) types for
a Midi Device Schema (as is the case when overriding a field). Xml Schema
parsers will generate an error for this condition which can be ignored.
Declare a Composite data type
A new custom data type
that specifies a union of other data types will provide this behavior.
Data type constraints
An Xml Schema facet
can be used to constrain the valid value range of a data type.
The following example
shows the constraints placed on a data byte in a SysEx
message.
<xs:simpleType name="midiSysExData"> <xs:restriction base="midiByte"> <xs:minInclusive
value="0" /> <xs:maxInclusive
value="127" /> </xs:restriction> </xs:simpleType> |
The following xml
facets are supported:
Xml Facet |
Description |
minInclusive |
Specifies the
minimal value for the data type. The value specified lies inside the valid
range. |
maxInclusive |
Specifies the
maximal value for the data type. The value specified lies inside the valid
range. |
minExclusive |
Specifies the
minimal value for the data type. The value specified lies outside the valid
range. |
maxExclusive |
Specifies the
maximal value for the data type. The value specified lies outside the valid
range. |
enumeration |
Specifies an exact
value that is valid for the data type. Multiple enumeration facets can be
used fro one data type. |
length |
Specifies the fixed
length of a data type. Only valid for data types deriving from midiComposite. |
Constant values
There are two ways to
declare constant values.
The following example
shows the fixed values for the SysEx begin- and
end-markers in a midi SysEx message.
<xs:complexType name="midiSysExMsg"> <xs:sequence> <xs:element name="SOX" type="mt:midiSysEx" fixed="240" /> <xs:element name="Body" type="midiSysExMsgAbstractBody" /> <xs:element name="EOX" type="mt:midiSysEx" fixed="247" /> </xs:sequence> </xs:complexType> |
Documenting a schema
You can use the
standard Xml Schema annotation tags to document the Midi Device Schema
declarations.
<xs:schema targetNamespace="…" xmlns:xs="http://www.w3.org/2001/XMLSchema"> <xs:annotation> <xs:documentation>The midiTypes schema represents the base types used in the midi device schema framework.</xs:documentation> </xs:annotation> <xs:simpleType name="midiByte"> <xs:annotation> <xs:documentation>midiByte is the base type for all (simple) midi data types. It represents an unsigned 8-bit byte.</xs:documentation> </xs:annotation> <xs:restriction base="xs:unsignedByte" /> </xs:simpleType> … </xs:schema> |
[Note: The type system
is still under development!]
The Midi Device Schema
type system is designed to allow a schema author to express the logical midi
System Exclusive fields of a device. All types inherit from the midiByte data type which represents an unsigned byte
(8-bits).
The following figure
displays the type hierarchy for the (simple) data types declared in the type
system. The lines represent inheritance and the arrows point to the derived
(more specific) type.
|
midiByte
Represent the base
data type for all other types but is not declared abstract and thus can be used
as any data type. The value represents the full 8 bits of an unsigned byte.
midiBit0-midiBit7
The midiBit0 till
midiBit7 data types each represent one bit of the 8-bit byte. midiBit0 represents the least significant bit and midiBit7
represents the most significant bit.
midiNull
The midiNull data type can be used for fillers or dummy fields.
The value defaults to 0 (zero) but can be set at a per field basis using the fixed attribute on the field
declaration. Fields that use this data type are not visible at a logical level.
When logical data is converted to physical midi data, these fields are inserted
automatically at their designated positions within the record.
midiSysExData
The midiSysExData represents a data byte inside a midi System
Exclusive message. This data type enforces that bit7 (most significant bit) is
always clear (not set) to comply with the midi specifications for System
Exclusive messages.
midiNibble
The midiNibble data type is abstract and cannot be used in a
field declaration. The data type is a logical indication that the data is
4-bits in length (a nibble) and is mainly used by the nibble-data converter
implementation.
midiLSN
The midiLSN represents the Least Significant Nibble (LSN) of a
data byte (lower 4 bits).
midiMSN
The midiMSN represents the Most Significant Nibble (MSN) of a
data byte (upper 4 bits).
Note that a
contradiction is hidden in this data type: The midiMSN
derives from midiSysExData (via midiNibble).
That type ensures that bit7 is always cleared. This means that bit7 of a midiMSN field is also always cleared.
midiComposite
The midiComposite is an abstract type that is used as a base
for logical values that span multiple physical bytes.
This type is not
intended to be derived by schema authors.
midiString
The midiString (derives from midiComposite)
represent multiple physical data bytes that make up one logical text string.
In order to use this
in a schema declaration one should derive a custom data type and specify the length facet for that custom type. The
string-data converter will use the specified length to read the text string.
The Type System also
contains (complex) record types (field structures). Checksum calculations for
instance can be implemented this way. These types are still under development
and too volatile to document here. These will be added as soon as the design is
somewhat more stable.
The current
functionality for these (complex) record base types include:
This section delves
into implementation details for these specifications. Some assumptions exist on
how the implementation should handle certain events. This description follows
the base architecture provided at the beginning of this document.
The schema object
model provides an in-memory representation of a Midi Device Schema declared in
an .xsd file (or other resource).
The most important
feature of this component is the ability of interpreting the xml in the schema
and compiling this into a fully linked object model.
These operations are
performed during the compile phase:
The following classes
could be expected to be in a Schema Object Model:
Class |
Description |
SchemaManager |
An entry point for
caching and loading schemas. Is used during schema compilation to resolve
imported types. |
Schema |
The root object for
a specific schema. Created based on an .xsd file
(or resource). Contains all declarations made in the Midi Device Schema. |
RecordType |
A representation of
a data record containing fields. |
DataType |
A description of a (custom)
data type. |
Field |
A description of a
field. Including Declaring RecordType and
Constraints. |
Constraint |
A description of a
type constraint. |
For each class, except
the SchemaManager, a collection or list class should
also be available.
(Refer to the
Microsoft .NET System.Xml.Schema namespace for
inspiration.)
Flattened Field List
A record type consists
of fields. Each field can either be a value (data type) or a sub-record (record
type). This produces a hierarchy of record types where data fields are leaves
in the tree. A flattened field list is a list of all fields effectively in a
record type. This list will only list data field (with data types) and is basically
the flattened structure of the system exclusive data.
A typical scenario for
the Data Object Model would be to transform data for a specific midi device. The
Midi Device Schema (class) from the SOM is interpreted and the data converter
factories are called to create data or group converters for all data and record
types. Once the converters are in-place the DOM can start to receive
transformation requests to transform from physical midi data to logical midi
data or visa versa.
It stands to reason
that the DOM supplies some sort of caching mechanism for the converters (they
are stateless) and a class to expose the data transformation services based on
a specific schema.
[TODO: explain
relation between schema and converters and converter hierarchy and converter
types (converter and group converter)]
The hierarchy of data
records described in the Schema
Object Model section also reflects back onto the data converters.
Data converters are created through registered converter factories. The factory
supports two create methods; one for creating a data converter for a data type
and the other for creating a group-converter for a record type. Group
converters are containers for other converters and can intercept the to-logical
or to-physical data transformation calls.
[TODO: explain the
implementation of ToLogical and ToPhysical
along with Carry]