Implementing Articulations in Digital Music Input through Keyboard and Smartphone Input

Interaction Design Oct 17, 2021


With the growing popularity of MIDI interfaces and electronic/digital music, we have entered an era wherein the computer becomes the major workstation for musicians as well, to compose, ideate, record, arrange and experiment with new sounds and music.

While most of the producers rely on external interfaces for inputting music into their workstations, we seek to imagine a budget-friendly method to provide producers with the same experience of inputting various music parameters through standard peripherals like keyboard and mouse.

What is MIDI?

MIDI is a technical standard that describes a communications protocol, digital interface, and electrical connectors that connect a wide variety of electronic musical instruments, computers, and related audio devices for playing, editing and recording music.

VSTs and DAWs

MIDI data is then read and translated to audio by plugin interfaces known as VSTs (Virtual Studio Technology).

VSTs integrate software synthesizers and effects units into Digital Audio Workstations (DAWs). VST and similar technologies use digital signal processing to simulate traditional recording studio hardware into software.

Existing MIDI controllers

The Effect of Articulations

Articulations are what make music expressive. Most DAWs at the moment allow you to input notes from a traditional keyboard, but the input of other articulations and features are mostly ignored and users are expected to edit those details manually later.


To understand the field of our project, we began by looking into various different methods of MIDI data input, most of which were in the form of dedicated hardware devices.

Inputting through Standard Peripherals

We mapped out all the current easily available hardware and their affordances and how we can associate music articulations to them. This helped us to set a broader understanding of the inputs, and how multiple inputs can be stringed together to create natural interactions to mimic musical instruments.

Our Design

Our design incorporates inputs from a smartphone and the PC QWERTY keyboard to give the users the power to create expressive music.

Binary/Digital input is mapped to the keyboard, while the phone becomes the source for all analog input.

Initial Prototype

We were able to develop a working Android application to test out our vision with the users. Though it did not match our intended design solutions, we were able to get a rough idea of inputting MIDI data through a smartphone while triggering notes from a keyboard.

Interface Design

We had 2 modes of binary input mapped to the keyboard which could be cycled through using the caps lock key. A Chord Mode to input chords and an individual notes mode to cater to inputting leads and melody lines. All other analog input was to be done through the smartphone interface.

User Testing

For our user testing, the users were given a walkthrough of our interface and technique and were told to experiment with the same while thinking aloud.

User Feedback and Proposed Solutions

For our user testing, the users were given a walkthrough of our interface and technique and were told to experiment with the same while thinking aloud.

  • Two-handed use can have a steep learning curve.
  • Inputs can be made in multiple cycles to provide a layer of intonations, instead of doing it all in one go.
  • Pitch bending will require relative positioning and will require the setup of a new origin after each use.
  • Velocity might be more suitable for accelerometer input.
  • Accelerometer mapping should match VST interface mapping.
  • Velocity and cutoff mapping require unergonomic hand movement if paired with Pitch bend input at the same time.

Updated Interface

We followed through with the suggestions made on the interface and remapped a few default settings to make the interactions more intuitive. The pitch bend slider would be used live while playing through on the keyboard, while all other inputs could be recorded on a second pass. The user would also be given an option to remap attributes as per his requirements, and would also be able to swipe to cycle through similar interfaces mapped to different parameters.


Rishabh Kumar

A product-oriented and grounded approach to design to create scalable and tangible solutions for organisations.

Great! You've successfully subscribed.
Great! Next, complete checkout for full access.
Welcome back! You've successfully signed in.
Success! Your account is fully activated, you now have access to all content.