Pixel to App
- People
- Christoph Csallner
- Don Dang
- Sabrina Haque
- Siva Natarajan Balasubramania
- Soumik Mohian
- Tony Tang
- Tuan Anh Nguyen
- Tuan Trinh
- Code
- Altify
- D2S2 drag 'n' drop screen search
- Doodle2App
- PSDoodle search via partial screen doodle
- REMAUI v2 reverse engineering mobile application user interfaces
- WnD / TpD search via text + doodle
- Data
- DoodleUINet
- Drawing interface used to collect DoodleUINet
Papers
- Altify: Shift-left icon alt-text for mobile app developers.
Proc. 23rd International Web for All Conference (W4A), 2026.
(Figshare)
-
Altify builds on IconDesc to shift accessibility tasks to earlier in the app development workflow, by generating high-quality icon alt-text for both finished and partially-completed app screens.
- D2S2: Drag 'n' drop mobile app screen search.
Proc. 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), Demonstrations Track, 2023.
(GitHub,
YouTube)
-
D2S2 is the first interactive drag-and-drop search engine for mobile app screenshots. D2S2 encodes text queries and the location, size, and icon type of the icons a user dragged on D2S2's UI. In a preliminary experiment D2S2 produced significantly more relevant search results than Google Image Search.
- Searching mobile app screens via text + doodle.
2023.
(GitHub)
-
TpD is the first interactive and iterative search system for mobile screens that incorporates both text and sketch input. TpD's source code, processing scripts, training data, and experimental results are freely available.
- PSDoodle: Fast app screen search via partial screen doodle.
Proc. 9th IEEE/ACM International Conference on Mobile Software Engineering and Systems (MOBILESoft), 2022.
(GitHub,
YouTube:
90 seconds,
15 minutes)
-
Existing screen search tools are either keyword-based (Google Image Search) or require the user to produce a sketch of a complete screen. PSDoodle is the first tool to allow interactive screen search via interactive freehand sketching.
- PSDoodle: Searching for app screens via interactive sketching.
Proc. 9th IEEE/ACM International Conference on Mobile Software Engineering and Systems (MOBILESoft), Tool Demos and Mobile Apps Track, 2022.
(GitHub,
YouTube:
90 seconds,
10 minutes)
-
This paper adds details on PSDoodle's architecture, deployment, and evaluation.
- Doodle2App: Native app code by freehand UI sketching.
Proc. 7th IEEE/ACM International Conference on Mobile Software
Engineering and Systems (MOBILESoft), Tool Demos and Mobile Apps Track,
2020.
(GitHub,
DoodleUINet data set,
GitHub to collect DoodleUINet,
slides)
-
We trained Doodle2App on thousands of (manually created)
freehand sketches of 20 common Android UI elements. The resulting
website allows users to sketch an Android screen. The website then
converts the sketched screen to Android code that is ready to compile
and run on stock Android phones. In technical terms: We pre-trained a
classifier on 2 million Google "Quick, Draw!" sketches and retrained the
classifier on thousands of sketches collected via Amazon Mechanical Turk.
In some sense the resulting tool is a modern version of SILK
("Sketching Interfaces Like Krazy").
-
P2A: A tool for converting pixels to animated mobile application user
interfaces.
Proc. 5th IEEE/ACM International Conference on Mobile Software
Engineering and Systems (MOBILESoft), 2018.
(slides)
-
P2A takes as input a set of screen design bitmaps
(e.g., screenshots of an Android or iPhone app) and converts them to
native app code (i.e., for Android), complete with inter-screen
transitions and in-screen animations. P2A is implemented on top
of REMAUI.
-
Reverse engineering mobile application user interfaces with REMAUI. Proc. 30th IEEE/ACM International Conference on Automated
Software Engineering (ASE), 2015.
(slides,
award)
-
When developing a mobile app (e.g., for Android or iOS), a
graphic designer typically designs the app's screens and hands
them to a programmer, who manually recreates the screen designs in
source code. REMAUI is the first technique for automating this
process end-to-end, from design drawings or screenshots to working
UI code that can be compiled and run on a mobile device.
Received an ACM SIGSOFT Distinguished Paper Award.