The following is a list of research projects I have been involved in, including art-creation tools, perception-based stylistic depiction, non-photorealistic rendering, and non-linear dimensionality reduction, amongst others. Within each topic, entries are ordered by date. For some of my past research, I've kept separate webpages. Many projects are accompanied by movies. The later movies are linked by their formats. Earlier movies often use an MPEG-4 codec from Microsoft, which I cannot make available on this site. Many recent MPEG-4 codecs can play them, though. If you want a very neat free player for almost any platform that natively supports a variety of codecs, download Video Lan Player.

 Products

PaintCan Logo

PaintCan app available for free on iOS! Download it today.

    With PaintCan you can create beautiful paintings from your favorite photographs within minutes.

PaintCan is a fun, easy way to paint great looking artwork from your photographs. It is based on my "Painting with Bob" research (to be added soon). Visit our webpage to learn more.

 Art-Creation Tools

WYSIWYG Stereo Painting
To appear in ACM Symposium on Interactive 3D Graphics and Games (I3D) 2013.
Yongjin Kim, Holger Winnemöller, Seungyong Lee


[PDF] [WWW] [BIB]

Abstract: Despite increasing popularity of stereo capture and display systems, creating stereo artwork remains a challenge. This paper presents a stereo painting system, which enables effective from-scratch creation of high-quality stereo artwork. A key concept of our system is a stereo layer, which is composed of two RGBAd (RGBA + depth) buffers. Stereo layers alleviate the need for fully formed representational 3D geometry required by most existing 3D painting systems, and allow for simple, essential depth specification. RGBAd buffers also provide scalability for complex scenes by minimizing the dependency of stereo painting updates on the scene complexity. For interaction with stereo layers, we present stereo paint and stereo depth brushes, which manipulate the photometric (RGBA) and depth buffers of a stereo layer, respectively. In our system, painting and depth manipulation operations can be performed in arbitrary order with real-time visual feedback, providing a flexible WYSIWYG workflow for stereo painting. Comments from artists and experimental results demonstrate that our system effectively aides in the creation of compelling stereo paintings.

Texture Design and Draping in 2D Images
(Proc. Eurographics Rendering Symposium) Computer Graphics Forum 28(4), 2009.
Holger Winnemöller, Alexandrina Orzan, Laurence Boissieux, Joëlle Thollot


[PDF] [WWW] [MOV] [BIB]

Abstract: We present a complete system for designing and manipulating regular or near-regular textures in 2D images. We place emphasis on supporting creative workflows that produce artwork from scratch. As such, our system provides tools to create, arrange, and manipulate textures in images with intuitive controls, and without requiring 3D modeling. Additionally, we ensure continued, non-destructive editability by expressing textures via a fully parametric descriptor. We demonstrate the suitability of our approach with numerous example images, created by an artist using our system, and we compare our proposed workflow with alternative 2D and 3D methods.

Diffusion Curves: A Vector Representation for Smooth-Shaded Images
ACM Transactions on Graphics (Proc. of SIGGRAPH'08), Volume 27, 2008.
Alexandrina Orzan, Adrien Bousseau, Holger Winnemöller, Pascal Barla, Joëlle Thollot, David Salesin


[PDF] [YouTube] [WWW] [BIB]

Abstract: We describe a new vector-based primitive for creating smooth-shaded images, called the diffusion curve. A diffusion curve partitions the space through which it is drawn, defining different colors on either side. These colors may vary smoothly along the curve. In addition, the sharpness of the color transition from one side of the curve to the other can be controlled. Given a set of diffusion curves, the final image is constructed by solving a Poisson equation whose constraints are specified by the set of gradients across all diffusion curves. Like all vector-based primitives, diffusion curves conveniently support a variety of operations, including geometry-based editing, keyframe animation, and ready stylization. Moreover, their representation is compact and inherently resolution-independent. We describe a GPU-based implementation for rendering images defined by a set of diffusion curves in realtime. We then demonstrate an interactive drawing system for allowing artists to create artworks using diffusion curves, either by drawing the curves in a freehand style, or by tracing existing imagery. The system is simple and intuitive: we show results created by artists after just a few minutes of instruction. Furthermore, we describe a completely automatic conversion process for taking an image and turning it into a set of diffusion curves that closely approximate the original image content.

 Perception-based Stylistic Depiction [top]

XDoG: An eXtended Difference-of-Gaussians Compendium including Advanced Image Stylization
Computers & Graphics, 36(6), pp. 740--753, 2012.
Holger Winnemöller, Sven C. Olsen, Jan Eric Kyprianidis


[PDF] [WWW] [Code] [BIB]

Abstract: Recent extensions to the standard difference-of-Gaussians (DoG) edge detection operator have rendered it less susceptible to noise and increased its aesthetic appeal. Despite these advances, the technical subtleties and stylistic potential of the DoG operator are often overlooked. This paper offers a detailed review of the DoG operator and its extensions, highlighting useful relationships to other image processing techniques. It also presents many new results spanning a variety of styles, including pencil-shading, pastel, hatching, and woodcut. Additionally, we demonstrate a range of subtle artistic effects, such as ghosting, speed-lines, negative edges, indication, and abstraction, all of which are obtained using an extended DoG formulation, or slight modifications thereof. In all cases, the visual quality achieved by the extended DoG operator is comparable to or better than those of systems dedicated to a single style.

XDoG: Advanced Image Stylization with eXtended Difference-of-Gaussians
Best Paper Award: Non-photorealism
Proceedings of NPAR 2011.

Holger Winnemöller


[PDF] [Presentation] [BIB]

Abstract: Recent extensions to the standard Difference-of-Gaussians (DoG) edge detection operator have rendered it less susceptible to noise and increased its aesthetic appeal for stylistic depiction applications. Despite these advances, the technical subtleties and stylistic potential of the DoG operator are often overlooked. This paper reviews the DoG operator, including recent improvements, and offers many new results spanning a variety of styles, including pencilshading, pastel, hatching, and binary black-and-white images. Additionally, we demonstrate a range of subtle artistic effects, such as ghosting, speed-lines, negative edges, indication, and abstraction, and we explain how all of these are obtained without, or only with slight modifications to an extended DoG formulation. In all cases, the visual quality achieved by the extended DoG operator is comparable to or better than those of systems dedicated to a single style.

Errata: Due to a production problem, Eq.6 does not correctly reflect the intention of Fig.4. The sign should be flipped. For the correct equation and a convenient re-formulation, please refer to the follow-up paper, above.

Using NPR to evaluate perceptual shape cues in dynamic environments
Proceedings of NPAR 2007.
Holger Winnemöller, David Feng, Sotori Suzuki, Bruce Gooch


[PDF] [XVID] [BIB] [3D Models (3DS)] [3D Models (OBJ)]

Abstract: We present a psychophysical experiment to determine the effectiveness of perceptual shape cues for rigidly moving objects in an interactive, highly dynamic task. We use standard non-photorealistic (NPR) techniques to carefully separate and study shape cues common to many rendering systems. Our experiment is simple to implement, engaging and intuitive for participants, and sensitive enough to detect significant differences between individual shape cues. We demonstrate our experimental design with a user study. In that study, participants are shown 16 moving objects, 4 of which are designated targets, rendered in different shape-from-X styles. Participants select targets projected onto a touch-sensitive table. We find that simple Lambertian shading offers the best shape cue in our user study, followed by contours and, lastly, texturing. Further results indicate that multiple shape cues should be used with care, as these may not behave additively.

Perceptually-motivated Non-Photorealistic Graphics
PhD Dissertation, Northwestern University, 2006.
Holger Winnemöller


[PDF] [BIB]

Abstract (abbr.): [...] This dissertation demonstrates the mutual beneficence of non-realistic computer graphics and perception with two rendering frameworks and accompanying psychophysical studies: (1) Inspired by low-level human perception, a novel image-based abstraction framework simplifies and enhances images to make them easier to understand and remember. (2) A non-realistic rendering framework generates isolated visual shape cues to study human perception of fast-moving objects. The first framework leverages perception to increase effectiveness of (non-realistic) images for visually-driven tasks, while the second framework uses non-realistic images to learn about task-specific perception, thus closing the loop. As instances of the bi-directional connections between perception and non-realistic imagery, the frameworks illustrate numerous benefits including effectiveness (e.g. better recognition of abstractions versus photographs), high performance (e.g. real-time image abstraction), and relevance (e.g. shape perception in non-impoverished conditions).

Real-Time Video Abstraction
ACM Transactions on Graphics, Proc. SIGGRAPH 2006.
Implementation Sketch at SIGGRAPH 2006.
Poster at EDGE Workshop 2006.

Holger Winnemöller, Sven C. Olsen, Bruce Gooch


[WWW] [PDF] [BIB] [PPT] [RECORDING] [SKETCH] [POSTER]

Abstract: We present an automatic, real-time video and image abstraction framework that abstracts imagery by modifying the contrast of visually important features, namely luminance and color opponency. We reduce contrast in low-contrast regions using an approximation to anisotropic diffusion, and artificially increase contrast in higher contrast regions with difference-of-Gaussian edges. The abstraction step is extensible and allows for artistic or data-driven control. Abstracted images can optionally be stylized using soft color quantization to create cartoon-like effects with good temporal coherence. Our framework design is highly parallel, allowing for a GPU-based, real-time implementation. We evaluate the effectiveness of our abstraction framework with a user-study and find that participants are faster at naming abstracted faces of known persons compared to photographs. Participants are also better at remembering abstracted images of arbitrary scenes in a memory task.

 Non-photorealistic Rendering [top]

NPR in the Wild
Image and Video-based Artistic Stylization, Computational Imaging and Vision Vol.42, pp.353--374, 2013.
Holger Winnemöller


[Preprint] [WWW] [BIB]

Abstract: During the early years of computer graphics, the results were arguably not as realistic as the intended goal set forth. However, it was not until sometime later that non-realism was accepted as a goal worthwhile pursuing. Since then, NPR has matured considerably and found uses in numerous applications, ranging from movies and television, production tools, and games, to novelty and casual creativity apps on mobile devices. This chapter presents examples from each of these categories within their historical and applied context.

Stylized Vector Art from 3D Models with Region Support
(Proc. Eurographics Rendering Symposium) Computer Graphics Forum 27(4), June 2008.
Elmar Eisemann, Holger Winnemöller, John C. Hart, David Salesin


[PDF] [WWW]

Abstract: We describe a rendering system that converts a 3D meshed model into the stylized 2D filled-region vector-art commonly found in clip-art libraries. To properly define filled regions, we analyze and combine accurate but jagged face-normal contours with smooth but inaccurate interpolated vertex normal contours, and construct a new smooth shadow contour that properly surrounds the actual jagged shadow contour. We decompose region definition into geometric and topological components, using machine precision for geometry processing and raster-precision to accelerate topological queries. We extend programmable stylization to simplify, smooth and stylize filled regions. The result renders 10K-face meshes into custom clip-art in seconds.

Forward Lean - Deriving Motion Illustrations from Video
ACM SIGGRAPH Asia 2008, Sketch Program
Marc Nienhaus, Holger Winnemöller, Jürgen Döllner, Bruce Gooch


[PDF] [MP4] [M4V] [3GP]

Abstract: Forward lean, aka affine shear, is a depiction technique that illustrates a moving object by shearing its image into the direction of its motion, so that the object resembles a leaning person overcoming inertia at the start of locomotion or fighting wind resistance [Thomas and Johnson 1995]. Traditionally, artist use forward lean to effectively represent the motion of an object in static images, e.g. in posters or in comic strips [Cutting 2002]. We detect and extract moving objects and their motion direction from a video sequence and then apply forward lean to the extracted object in order to illustrate the objects' motions in a single static image.

Rendering Optimisations for Stylised Sketching
Proc. ACM Afrigraph 2003
Holger Winnemöller, Shaun Bangay


[WWW] [PDF] [BIB]
[AVI] [AVI] [AVI] [AVI]

Abstract: We present work that specifically pertains to the rendering stage of stylised, non-photorealistic sketching. While a substantial body of work has been published on geometric optimisations, surface topologies, space-algorithms and natural media simulation, rendering-specific issues are rarely discussed in-depth even though they are often acknowledged. We investigate the most common stylised sketching approaches and identify possible rendering optimisations. In particular, we define uncertainty-functions, which are used to describe a human-error component, discuss how these pertain to geometric perturbation and textured silhouette sketching and explain how they can be cached to improve performance. Temporal coherence, which poses a problem for textured silhouette sketching, is addressed by means of an easily computed visibility-function. Lastly, we produce an effective yet surprisingly simple solution to seamless hatching, which commonly presents a large computational overhead, by using 3-D textures in a novel fashion. All our optimisations are cost-effective, easy to implement and work in conjunction with most existing algorithms.

Geometric Approximations Towards Free Specular Comic Shading
Computer Graphics Forum 21(3), Proc. Eurographics 2002.
Holger Winnemöller, Shaun Bangay


[WWW] [PDF] [BIB]
[AVI] [AVI]

Abstract: We extend the standard solution to comic rendering with a comic-style specular component. In order to minimise the computational overhead associated with this extension, we introduce two optimising approximations: the perspective correction angle and the vertex face-orientation measure. Even though both of these optimisations are generally applicable, they are especially well-suited for applications where a physically correct lighting simulation is not required. With the optimisations in place, we achieve performances comparable to the standard solution. As our approximations favour large models, we even outperform the standard approach for models consisting of 10,000 triangles or more, which we can render exceeding 40 frames per second, including the specular component.

Implementing Non-Photorealistic Rendering Enhancements with Real-Time Performance
Masters Thesis, Rhodes University, 2002.
Holger Winnemöller


[WWW] [PDF] [BIB]

Abstract (abbr.): We describe quality and performance enhancements, which work in real-time, to all well-known Non-photorealistic (NPR) rendering styles for use in an interactive context. These include Comic rendering, Sketch rendering, Hatching and Painterly rendering, but we also attempt and justify a widening of the established definition of what is considered NPR. In the individual Chapters, we identify typical stylistic elements of the different NPR styles. We list problems that need to be solved in order to implement the various renderers. Standard solutions available in the literature are introduced and in all cases extended and optimised. In particular, we extend the lighting model of the comic renderer to include a specular component and introduce multiple inter-related but independent geometric approximations which greatly improve rendering performance. We implement two completely different solutions to random perturbation sketching, solve temporal coherence issues for coal sketching and find an unexpected use for 3D textures to implement hatch-shading. Textured brushes of painterly rendering are extended by properties such as stroke-direction and texture, motion, paint capacity, opacity and emission, making them more flexible and versatile. Brushes are also provided with a minimal amount of intelligence, so that they can help in maximising screen coverage of brushes [...]

 Non-linear Dimensionality Reduction (NLDR) [top]


[WWW] [PDF] [BIB]

Temporal Segmentation and Visualization of High-Dimensional Data using
Non-linear Dimensionality Reduction

In Submission, 2006.
Holger Winnemöller, Bruce Gooch

Abstract: We propose a visual analysis framework for inspecting and synthesizing temporally-dependant, high-dimensional data by combining the output of two non-linear dimensionality reduction techniques. Our framework segments a timeline by clustering data-frames using Locally Linear Embedding (LLE). The temporally annotated clusters are then visualized in up to three dimensions using an Isomap-based graph layout. These graphs contain a rich set of scene relevant information and can be used (1) to discover spatially and temporally under- and over-sampled areas, (2) to plan traversal paths for data synthesis, and (3) as input to surveillance and summarization algorithms.


[PDF1] [PDF2] [BIB] [XVID]

Lightwaving: Estimating Light Positions from Photographs Alone
Computer Graphics Forum 24(3), Proc. Eurographics 2005. (PDF1)
Sketch at ACM SIGGRAPH 2005. (PDF2)

Holger Winnemöller, Ankit Mohan, Jack Tumblin, Bruce Gooch

Abstract: We present an algorithm to automatically estimate three-dimensional light positions from an unordered set of images. We collect images using a single stationary camera while manually moving a light source around an object. Rather than measuring light positions directly, the algorithm extracts a three-dimensional manifold of positions from the images using dimensionality reduction techniques. This obviates the need for calibration and specialized equipment, making our approach inexpensive, portable and applicable to objects of almost any size. We demonstrate our results using image-based relighting applications.

 Other Work [top]

Creating collections with automatic suggestions and example-based refinement
UIST '10. ACM, New York, NY, 249-258
Adrian Secord, Holger Winnemöller, Wilmot Li, Mira Dontcheva


[PDF] [ACM/DOI] [BIB]

Abstract: To create collections, like music playlists from personal media libraries, users today typically do one of two things. They either manually select items one-by-one, which can be time consuming, or they use an example-based recommendation system to automatically generate a collection. While such automatic engines are convenient, they offer the user limited control over how items are selected. Based on prior research and our own observations of existing practices, we propose a semi-automatic interface for creating collections that combines automatic suggestions with manual refinement tools. Our system includes a keyword query interface for specifying high-level collection preferences (e.g., "some rock, no Madonna, lots of U2,") as well as three example-based collection refinement techniques: 1) a suggestion widget for adding new items in-place in the context of the collection; 2) a mechanism for exploring alternatives for one or more collection items; and 3) a two-pane linked interface that helps users browse their libraries based on any selected collection item. We demonstrate our approach with two applications. SongSelect helps users create music playlists, and PhotoSelect helps users select photos for sharing. Initial user feedback is positive and confirms the need for semi-automated tools that give users control over automatically created collections.

Automatic HDR Compositing
Unpublished (2004).
Holger Winnemöller, Ankit Mohan, Jack Tumblin


[PDF]

Abstract: Many scenes a photographer might encounter contain bright lights and dark, shaded regions. The light-intensities of such a scene span a HIGH DYNAMIC RANGE (HDR). Most cameras can only capture a small subset of this range, leaving all other areas either underexposed (black) or overexposed (white). To capture all of the dynamic range of a scene we can take a set of photographs with different exposure settings (shutter speeds) and combine the resulting images. We are exploring a novel way of automating this recombination process. Our method uses changes in intensities (gradients) instead of intensities themselves.


[PDF] [WWW] [BIB]

Testing Effects of Color Constancy for Images displayed on CRT Devices
Technical Report, Department of Computer Science, University of Cape Town, 2003.
Holger Winnemöller

Abstract: In this report we examine the effects of color constancy for images displayed on computer monitors, particularly CRT devices. The motivation for this work is based upon Edwin Land's Retinex Theory which attempts to explain color constancy in human perception. Using Land's theory, we can create images with a severely undersampled color space (down to a full omission of one color component) that are still appropriately interpreted by the human observer. Possible applications of this are compression schemes as well as image enhancement techniques, some of which are already known. We present a simple web-based testing scheme to verify the viability of displaying duo-chrome (e.g. extreme color undersampling) images on CRT devices in different environmental settings. Some results and possible interpretations of these results are also presented.


[PDF] [WWW]

A Review of Zeki's: A Vision of the Brain
Unpublished, for personal use only (2002).
Holger Winnemöller

Note: In order to learn about the different approaches to perception (Gestalt, Ecological, Neurological, etc.) I studied various books and wrote reviews/summaries of them. The book I most carefully studied was Semir Zeki's "A Vision of the Brain", which gives a very intricate overview of the different cortical areas, their alleged functions and connectivity. It's a fascinating read for anyone interested in how the brain works - and, sometimes even more interestingly, also describes cases where the brain doesn't work. This report summarizes and distills the key elements of Zeki's book. I present no new work, but do, where applicable, impose my own considerations and cross-reference related work. The structure of this report is not bound to Zeki's work, as my focus lies primarily in human perception as it pertains to processes that can be modeled or otherwise replicated or utilized in a (visual) computing environment.

Disclaimer: Much of the document I have prepared is copied verbatim (using a super-efficient digital camera-based OCR setup I devised for this purpose). All images, unless states otherwise, are taken from Zeki's book or are reproduced based on images from that book. While I have obtained Prof. Zeki's permission for this, I am still awaiting response from the publisher. In the meantime, this report is intended for personal use only and shall not be published or distributed in any form whatsoever.


[PDF] [BIB]
[AVI] [AVI] [AVI]

Super-realistic Rendering using Real-Time Tweening
South African Telecom. Networks & Applications Conf., SATNAC 2001.
Holger Winnemöller, Shaun Bangay

Abstract: The realism of contemporary computer graphics (and especially Virtual Reality {VR}) is limited by the great computational cost of rendering objects of appropriate complexity with convincing lighting and surface effects. We introduce a framework that allows rendering of objects in true photographic quality using tweening. The simple but effective design of our system allows us not only to perform the necessary operations in real-time on standard hardware, but also achieve other effects like morphing. Furthermore, it is shown how our system can be gainfully employed in non-VR contexts like extreme low-bandwidth video-conferencing and others.

Practical Gesture Recognition for Controlling Virtual Environments
Honours Thesis, Rhodes University, 1999.
Holger Winnemöller


[WWW] [PDF] [BIB]

Abstract: This thesis takes a practical approach to gesture recognition. Emphasis is placed on applicability and usability. In order to discuss practical aspects of gesture recognition, a theoretical 3-D modeling application is examined. Two different gesture recognition systems are introduced, extended and compared in an extensive field-test. These systems both have the characteristics of being computationally simple (fast, little demands on resources) while remaining reliable and extendable. Recognition- and success-rates in the high 90%'s can be achieved. Learning and adaptation to new gestures is automated successfully for both systems. Useful results concerning the choice of gestures, features and other system-relevant variables are obtained. Problems with general view-dependent interaction and modeling-specific vertex-selection are discussed and solutions implemented.