RC Surface

Q: Can I map from the phone?

Yes. MAP, pick a control, then Bind or Trigger Note.

Any device with a browser becomes a performance controller for Ableton Live. Your machine serves the surface on the local network and the phone just opens a link.

Live 12.4.5+ Suite · iOS 15.4+ Safari or Chromium on Android · held in landscape · local network, nothing installed on the phone.

Published releases Source & licence

2.0

Signal chain — SDK control, optional OSC clock

┌─ NODE 0 ─────────────────────────────────┐
│ PHONE BROWSER          TOUCH IMU MIC CAM │◀─┐
│ PERF · MIX · SNP · SNS · AUD · VID · MAP │  │
└────────────────────┬─────────────────────┘  │
       HOP 1  ▼  HTTPS · LOCAL CA · WSS       │
┌─ NODE 1 ─────────────────────────────────┐  │
│ RC SURFACE SERVER                 v1.0.0 │  │
│ SMOOTH · CURVE EVAL · MORPH · SNAPSHOT   │  │
└────────────────────┬─────────────────────┘  │
       HOP 2  ▼  EXTENSIONS SDK · LOCAL       │
┌─ NODE 2 ─────────────────────────────────┐  │
│ EXTENSIONS SDK          LOCAL HOST API   │  │
│ PARAMETERS · MIXER · TARGET CATALOG      │  │
└────────────────────┬─────────────────────┘  │
       HOP 3  ▼  LIVE OBJECT MODEL · IN PROC  │
┌─ NODE 3 ─────────────────────────────────┐  │
│ ABLETON LIVE 12.4.5                SUITE ├──┘
│ SESSION · DEVICES · MIXER · TRANSPORT    │
└──────────────────────────────────────────┘
▼ CONTROL, OUTBOUND      ◀ STATE, INBOUND

2.1 · Clock source

Mode	Needs	What you get
AbletonOSC	AbletonOSC	Beat-accurate: metronome, subdivisions, swing, phase.
SDK	nothing	BPM simulator off Live's tempo. No beat phase.
Free	nothing	Internal clock. Not locked to Live's tempo.

AbletonOSC is needed only for Deep Sync, and installs separately as a Control Surface with UDP 11000/11001 free. Without it everything else works. Long-press SYNC for clock source, subdivisions, phase, swing and shapes.

3.0

Surface map — figs. 1 to 7

PERF MIX SNP SNS AUD VID MAP

Fig. 1 PERF — the layout as built Scale none Proj. plan

N	Region	Qty	What it is
1	Page tabs	6	PERF, MIX, SNP, SNS, AUD, VID. MAP overlays the current page; CFG shows on PERF and MIX.
2	Pad grid	12	Four across, three down. Behaviour comes from the pad mode, not the pad.
3	Pad mode	4	A momentary, B hold, C toggle, D burst. Sets all twelve; CFG overrides one pad.
4	XY pads	2	Captioned `XY 1` and `XY 2 (Physics)`. Each axis maps on its own.
5	LFOs	4	L1 to L4. Hz in FREE, note values in SYNC. Ceilings: sine 4 Hz, square 12 Hz. Shape in CFG.
6	Stutters	4	S1 to S4. Horizontal speed, vertical amplitude. Up to 15 Hz including ratchet.
7	PERF UTIL	6	CAP captures, slots 1 to 4 recall, OFF exits.
8	Live panel	7	BPM, transport, TRN, MAP, CFG.
9	Sync block	4	SYNC, settings, STAGE; CALIBRATE on SNS/AUD/VID.

Fig. 2 MIX — eight knobs and eight faders Scale none Proj. plan

N	Region	Qty	What it is
1	Page tabs	6	PERF, MIX, SNP, SNS, AUD, VID. MAP overlays the current page; CFG shows on PERF and MIX.
2	Knobs	8	`knob-1..8`, four across by two down. Relative vertical drag.
3	Faders	8	`fader-1..8`, one row. The thumb moves exactly as far as the finger.
8	Live panel	7	BPM, transport, TRN, MAP, CFG.
9	Sync block	4	SYNC, settings, STAGE; CALIBRATE on SNS/AUD/VID.

Fig. 3 SNP — eight stored states and vector morph Scale none Proj. plan

N	Region	Qty	What it is
1	Page tabs	6	PERF, MIX, SNP, SNS, AUD, VID. MAP overlays the current page; CFG shows on PERF and MIX.
2	Morph slots	8	Whole controller states in a four-by-two grid; recall or morph to one slot.
3	Performance controls	3	`CAPTURE`, `CLEAR SLOTS`, transition 0.1–5.0 s or synced, and Grid/Vector.
4	Vector morph pad	1	Blends slots 1 TL, 2 TR, 3 BL and 4 BR at once, weighted by the point.
8	Live panel	7	BPM, transport, TRN, MAP, CFG.
9	Sync block	4	SYNC, settings, STAGE; CALIBRATE on SNS/AUD/VID.

Fig. 4 SNS — motion and orientation axes Scale none Proj. plan

N	Region	Qty	What it is
1	Page tabs	6	PERF, MIX, SNP, SNS, AUD, VID. MAP overlays the current page; CFG shows on PERF and MIX.
2	Motion	6 ax	`GX GY GZ` angular rate, `AX AY AZ` acceleration, signed around a centre line.
3	Orientation	3 ax	Yaw, pitch and roll as numeric angles.
4	Local view	2	Shows or hides the two banks on this phone only; the sensors keep running.
5	Axis key	3 ax	Ties the channel names to the physical phone, so tilting it becomes a readable gesture.
8	Live panel	7	BPM, transport, TRN, MAP, CFG.
9	Sync block	4	SYNC, settings, STAGE; CALIBRATE on SNS/AUD/VID.

Fig. 5 AUD — audio input and twelve descriptors Scale none Proj. plan

N	Region	Qty	What it is
1	Page tabs	6	PERF, MIX, SNP, SNS, AUD, VID. MAP overlays the current page; CFG shows on PERF and MIX.
2	Audio input	1	Pick the browser default or a permitted input, then enable capture. It never starts by itself.
3	Input level	1	Input RMS, normalized 0..1. Not perceived loudness.
4	Signal history	1	Live 2.5-second history: Amplitude, Attacks, Tone, Texture, Bands or All.
5	Detector knobs	10	SENS, RELEASE, CURVE, group GAIN and SMOOTH. Note divisions in SYNC, milliseconds in FREE.
6	Analysis window	1	WINDOW x1/x2/x4 trades spectral detail for response time.
7	Built-in audio detectors	12	Twelve 0..1 mapping sources: sensor.audio.transient, sensor.audio.kick, sensor.audio.snare, sensor.audio.brightness, sensor.audio.centroid, sensor.audio.flux, sensor.audio.flatness, sensor.audio.spread, sensor.audio.rolloff, sensor.audio.low, sensor.audio.mid, sensor.audio.high.
8	Live panel	7	BPM, transport, TRN, MAP, CFG.
9	Sync block	4	SYNC, settings, STAGE; CALIBRATE on SNS/AUD/VID.

Fig. 6 VID — camera, learned poses, detectors Scale none Proj. plan

N	Region	Qty	What it is
1	Page tabs	6	PERF, MIX, SNP, SNS, AUD, VID. MAP overlays the current page; CFG shows on PERF and MIX.
2	Camera	1	On/off, confidence, recognition mode, and the live preview.
3	Learned poses	3	G1, G2 and G3, three examples each: `CAP`, `TEST`, `DEL LAST`, `CLR`. Recognition emits a momentary 0→1.
4	Built-in detectors	4	Open, Fist, Pinch and Victory, opt-in and untrained. Rotation belongs to Victory.
5	Direct MAP / CLUTCH	2	MAP X/Y/Z follows the hand; CLUTCH X/Y/Z is relative and freezes when the pinch opens.
6	Camera diagnostics	2	PALM size and signed FACE stay local to the preview; they are not mapping controls.
8	Live panel	7	BPM, transport, TRN, MAP, CFG.
9	Sync block	4	SYNC, settings, STAGE; CALIBRATE on SNS/AUD/VID.

Fig. 7 MAP — armed strip, then target picker Scale none Proj. plan

N	Region	Qty	What it is
1	Underlying page	1	MAP is an overlay, not a tab. The page underneath stays visible until a control is picked.
2	Armed strip	1	Shows only `MAP ARMED / TAP A CONTROL / DONE`, and goes once a control is tapped.
3	Editing overlay	1	A 400 px right-side pane owns the mapping editor.
4	Target filter	1	Searches target names; `Selected in Live` narrows to the current track and device.
5	Hierarchy	3 levels	Track → Device → Parameter, plus Song/Main/Master and Return groups.
6	Live selection shortcut	1	Uses Live's selected track and device as a filter; it picks no parameter for you.
8	Live panel	7	BPM, transport, TRN, MAP, CFG.
9	Sync block	4	SYNC, settings, STAGE; CALIBRATE on SNS/AUD/VID.

Every control on these plans is listed in 4.0.

.surface-tabs{display:none} .surface-deck{display:block} .surface-sheet{visibility:visible;opacity:1;pointer-events:auto} .surface-sheet + .surface-sheet{margin-top:30px}

4.0

Controls — inventory, header, families

Control inventory — 18 rows

CONTROL ROW ON PHONE
    │
    ├─ SDK ─── WSS ─── RC SURFACE SERVER ─── EXTENSIONS SDK ─── LIVE PARAMETER
    │
    ├─ MIDI ── WSS ─── RC SURFACE SERVER ─── NOTE ────────── LIVE MIDI HOST
    │
    └─ local ─────── PHONE STATE ONLY
                      └─ never leaves the phone · cannot map to Live

N	Module	Qty	Page	Sends	Target in Live
00	Performance pads	12	PERF	MIDI	Clip, scene or a trigger note
01	Pad modes A–D	4	PERF	local	Pad behaviour; CFG overrides it per pad
02	XY pads	2	PERF	SDK	Two parameters per pad
03	LFOs L1–L4	4	PERF	SDK	Any mapped parameter
04	Stutters S1–S4	4	PERF	SDK	Any mapped parameter
05	PERF UTIL column	6	PERF	local	CAP, slots 1–4, OFF
06	Knobs	8	MIX	SDK	Device parameter
07	Faders	8	MIX	SDK	Track volume
08	Morph slots	8	SNP	SDK	A whole controller state
09	Transition time	1	SNP	local	Free 0.1–5.0 s, or Sync 16 beats to 1/16
0A	Vector morph pad	1	SNP	SDK	Blends slots 1–4 by corner
0B	Motion	6 ax	SNS	SDK	GX GY GZ, AX AY AZ
0C	Orientation	3 ax	SNS	SDK	Yaw, pitch, roll
0D	Audio input	1	AUD	SDK	RMS, envelope, gate, attack, plus twelve descriptors
0E	Single-hand vision	1	VID	SDK	`sensor.vision.x`, `.y`, `.z`; Pinch Clutch `sensor.vision.pinch_x`, `_y`, `_z`; Open / Fist / Pinch / Victory; learned G1–G3
0F	Map picker	3 surfaces	MAP	local	Phone overlay, panel Mapping tab and the admin window
10	Header live panel	7	header	SDK	BPM, transport, TRN, MAP, CFG
11	Header sync block	4	header	SDK	SYNC, settings, STAGE, CALIBRATE

SDK rows map numbers to Live parameters. MIDI needs the optional Receiver. Local controls never leave the browser.

The header — on every page

TRN

Full-screen transport overlay: locator list, search, and the SYNCED / SDK / FREE readout.

MAP

Opens the mapping picker over the current page.

CFG

Per-control settings: pad mode, LFO shape, stutter mode, XY 2 physics. PERF and MIX only.

SYNC

Toggles SYNC / FREE. Long-press for Deep Sync Settings, 2.1.

CALIBRATE

Sets the neutral reference for the page: posture on SNS, level on AUD, lighting on VID.

STAGE

Locks the surface to the current page.

What each one does — from touch to Live

01Performance 02Continuous controls 03Snapshots 04Motion & audio 05Vision 06Mapping & MIDI

Pads, LFOs, stutters

A returns to zero, B keeps what you leave, C latches, D fires one burst — set per pad in CFG. LFOs and stutters share the clock: locked to Live's beat in SYNC, free-running in FREE.

VALUE ↑TIME →TOUCH = ▼ · UP = △

A · RELEASE

touchup

drag sets levelUP → immediate zero

B · HOLD

SAVEDKEEP

drag edits levelUP → hold new value

C · TOGGLE

TAP ONTAP OFF

two-touch latchhold + drag edits ON

D · BURST

70 ms attack450 ms release

one-shot A–Rdrag peak .15–1 · length follows the grid in SYNC

BEAT

1234

SYNC / FREErepeat at beat division

XY, knobs, faders

XY 2 coasts and rebounds after release; XY 1 stays where you leave it. Knobs and faders track the finger from the current value and never jump to the touch point.

1 · RELATIVE VERTICAL DRAGKNOB · FADER

The touch point is never the value

value

same
length

finger

At touch-down dy is 0, so nothing moves. The knob crosses its range in 150 px; a fader travels as far as the finger.

value = start + dy / rangePx KNOB 150 · MACRO KNOB 220 · FADER 150

2 · PHYSICSXY 2

Release does not stop it

travelledwhere it keeps goingpad edge · rebounds at ×0.75

15-POINT TRAIL v ×= 0.988 each frame · stops under 0.01

Snapshots and vector morph

Capture stores every mapped control into one of eight slots. Recall interpolates to it over a set time. Vector XY blends slots 1–4 by finger position.

STORE A STATE · RETURN OVER TIME · BLEND BETWEEN FOUR STATES

CAPTURE / RECALL

LIVE STATE

VOL

CUT

SEND

PAN

CAP

EIGHT SLOTS

12345678

CURRENTSLOT 3

RECALL = timed interpolation · Free 0.1–5.0 s or Sync in beats

VECTOR MORPH

BLENDED OUTPUT

A .12

B .54

C .08

D .26

four corner states → weighted blend

Motion, orientation, audio

CALIBRATE makes the current posture the reference. AUD turns sound into control: level, envelope, gate, attacks and twelve descriptors, graphed over the last 2.5 seconds.

MOTION SETS A REFERENCE · AUDIO TURNS SOUND INTO STABLE CONTROL

MOTION

CALIBRATE

AX AY AZ · GX GY GZ

AUDIO INPUTLIVE

RMS0.42

ENV0.38

SIGNAL / 2.5sLIVE

RMSENV

SENS0.65

band risedrag ↑↓

RELEASE45 ms

pulse tail10–500

WINDOW · 1024 @48kx1x2x4

SYNC follows Live BPM from 1/128 to 1/1, triplets and dotted included; FREE keeps its own milliseconds. Low, mid and high are K-weighted loudness (ITU-R BS.1770), in LU.

Built-in audio detectors

ATTACKS

TRANSIENT0.82

KICK0.64

SNARE0.10

TONE

BRIGHTNESS0.36

CENTROID3000 Hz

ROLLOFF 95%5000 Hz

TEXTURE

FLUX0.25

FLATNESS0.08

SPREAD3000 Hz

BANDS

LOW0.24

MID0.18

HIGH0.09

cyan = instantaneous RMS · green = smoothed envelope

Vision: hand, poses, clutch

Hand X/Y/Z is continuous. A held pinch engages the clutch and releasing it freezes the value. G1–G3 and the four built-in detectors are intentional outputs; PALM and FACE remain local camera diagnostics.

PINCH CLUTCH — A LOOP YOU CAN LEAVE AND COME BACK TO

1 · ENGAGE

the two tips touchcontact rises past .75the clutch takes hold

2 · MOVE

the hand travels in spaceX, Y and Z follow itfrom where you started

3 · RELEASE

the fingers partcontact drops below .55the value stays put

PINCH AGAIN, ANY TIME, ANYWHERE — IT RESUMES FROM THE SAVED VALUE

CONTACT = gap between landmark 4 and landmark 8, divided by palm size ARM at .75 · RELEASE below .55 · latched in between

sensor.vision.pinch_x · _y · _z relative, never absolute

MAP, MIDI, and mapping surfaces

Phone MAP and panel Mappings bind a control to a Live parameter or a trigger note, with curve, range, takeover and idle settings.

ONE SIGNAL PATH · SOURCE DOMAIN → 0..1 SHAPE → TARGET DOMAIN

SOURCE

sensor.audio.centroid

range you set

MEASURED SENSOR DOMAIN

TRANSFORM DESKINPUT → OUTPUT

NORMALIZE

inMin – inMax → 0..1

CLAMP0..1

SOURCE RANGE

RESPONSE CURVE

LINEAR · EXPONENTIAL · LOGARITHMIC · S-CURVE

TARGET RANGE

Song Tempo

20–300 BPM

184 BPM

SCALED LIVE PARAMETER

POLICYTAKEOVERSCALEPICKUPHOLDSAFE LOSSRELEASESMOOTHING

ACTION BINDTRIGGER NOTE LIVE PARAMETER / MIDI

.control-tabs{display:none} .control-layout,.control-deck{display:block} .control-sheet{visibility:visible;opacity:1;pointer-events:auto} .control-sheet + .control-sheet{margin-top:20px}

5.0

Situations that make sense — and the mapping

Scenario A · two performers

A second pair of hands over one Live set

A colleague opens the phone and shapes the return effects while the soloist keeps playing.

xy-1.x→distortion Dry/Wetxy-1.y→reverb send

Scenario B · arrangement change

Four corners for four sections

Intro, verse, chorus and outro in SNP slots 1–4, dragged between corners in Vector XY.

knob-1→Auto Filter cutofffader-1→reverb return

Scenario C · pinch clutch

Push a filter in the air

Standing, no table. Pinch with the palm to the camera, move the hand, open the fingers.

sensor.vision.pinch_z→Auto Filter cutoff

Scenario D · instrument motion

The body supplies the modulation

A phone strapped to a guitar: the turn during a swell becomes a rotation-rate gesture.

sensor.motion.gx→Echo feedback

Scenario E · acoustic input

Volume opens the effect

A phone near the microphone: louder phrases open the filter, silence closes it.

sensor.audio.envelope→Saturator Drive

6.0

Installation — to a mapped parameter

Clip 1 Install and first pairing, narrated mp4

Install the extension

Double-click the .ablx from Published releases to request installation in Live.

Open the panel

In Live, open Extensions → RC Surface → Show panel. It shows the QR code, the connection status and the real URL if a port is taken — the link drawn in 2.0.

Connect the phone

Scan the QR code, accept the self-signed certificate once, and hold the phone in landscape.

Arm the sensors

Enable Motion, Audio or Vision as needed. Raw camera and microphone never leave the phone — only numeric values do.

Map a control to Live

Tap MAP, pick a highlighted control, then bind it to a Live parameter or a trigger note. Notes need the bundled RC-Midi-Receiver.amxd on the target MIDI track: Live cannot insert Max for Live devices through the SDK.

Optional: Deep Sync

Install AbletonOSC as a Control Surface with UDP 11000 and 11001 free, then long-press SYNC to pick the clock — see 2.1.

7.0

When it does not work

Troubleshooting

Phone cannot find the host

Same Wi-Fi. Live allowed through the firewall. Rescan the current QR.

AbletonOSC not detected

Install it separately as a Live Control Surface.

Vision will not start

Accept the certificate, grant camera, reload.

Audio is silent

Grant microphone, then tap the audio control again.

No Extensions menu in Live

Check the Live 12.4.5+ Suite Extensions host is enabled, then restart Live.

Frequently asked

Which Live versions?

12.4.5+ Suite (Beta).

Android and iOS?

iOS 15.4+ Safari, or Chromium on Android.

Can I map from the phone?

Yes. MAP, pick a control, then Bind or Trigger Note.

What about latency?

Controls are coalesced at 30 Hz; audio has its own bounded path. Measure end to end on your network.

Camera and microphone?

Processed inside the phone's browser. Never uploaded.

How mature is it?

v1.0.0, smoke-tested in Live 12.4.5+ Suite (Beta) with a physical phone.

8.0

Reference documents

Installation Guide

Read INSTALL.md

User Manual

Read USER-GUIDE.md

Customization & Dev

Read CUSTOMIZATION.md

FAQ

Read FAQ.md

Security Threat Model

Read SECURITY.md

Privacy Policy

Read PRIVACY.md