international · peer reviewed · open access

Library Student Journal, 2007

Library Student Journal: Developing a Shape-and-Composition CBIR Thesaurus for the Traditional Chinese Landscape.

Developing a Shape-and-Composition CBIR Thesaurus for the Traditional Chinese Landscape1

Tang Li
University of Maryland
College of Information Science
College Park, MD, United States

Library Student Journal,
July 2007

Editor's Note

This paper has been awarded the 2007 Gerd Muehsam Award, sponsored by the Art Libraries Society of North America (ARLIS/NA).

Abstract

In the past decade, content-based image retrieval (CBIR) has been investigated extensively. Current research has suggested that the two elemental issues in CBIR, feature extraction and similarity measures, tend to be domain-specific. This paper develops a shape-and-composition CBIR thesaurus for Chinese landscape paintings dating from the Song to Qing periods (960-1911). The features were extracted from studying approximately 1,000 Chinese landscape paintings. The thesaurus emphasizes discrimination among object types in order to improve retrieval of relevant images. Therefore, it adopts not only basic shapes but also line and shape combinations. Furthermore, special shapes are developed for those object types that are either unique to Chinese art and culture, or are a peculiar shape that cannot easily be abstracted into basic shapes. Although it is domain-specific, the approach of developing and classifying the thesaurus may be applicable to CBIR of non-Chinese art images and CBIR in general.

Introduction

This paper develops a shape-and-composition content-based image retrieval (CBIR) thesaurus for Chinese landscape paintings dating from the Song to Qing dynasties (AD 960-1911). The thesaurus emphasizes discrimination among object types in order to improve retrieval of relevant images. It adopts not only basic shapes but also line and shape combinations. Special shapes represent object types that are either unique to Chinese art and culture, or are a peculiar shape that cannot easily be abstracted into basic shapes. Although it is domain-specific, the approach of developing and classifying the thesaurus may be applicable to CBIR of non-Chinese art images and CBIR in general.

Qing dynasty hanging scroll painting
Wooded Mountains at Dusk, Qing dynasty (1644-1911)
(c) The Metropolitan Museum of Art
www.metmuseum.org
Dated 1666; Kuncan (Chinese, 1612-1673); hanging scroll; ink and color on paper; 49 ¾ x 23 7/8 in. (126.2 x 60.6 cm.); inscribed by the artist. Bequest of John M. Crawford Jr., 1988.

Content-Based Image Retrieval (CBIR) vs. Text-Based Image Retrieval (TBIR)

Two major approaches have been identified in image indexing and retrieval: text-based (descriptor-based) and content-based. Text-based image retrieval (TBIR) has been widely adopted in the cataloging and indexing of image collections in libraries, museums, and archives, relying on manually assigned text descriptors to retrieve relevant images. More recently, automatic assignment of text attributes to images has been developed by extracting terms from captions and descriptions. Because it is based on traditional text Information Retrieval (IR) techniques, the TBIR system seems relatively easy to develop and use. However, as both Yang (2004) and Graham (2004) point out, manual assignment of text annotations to images is time-consuming and expensive. Further, it may be impossible to automate text assignment if no description accompanies an image. Moreover, although text-based image descriptors may seem comprehensive and objective, they are inevitably a partial and perhaps inadequate representation of visual information and are therefore subject to individual interpretation. Image catalogers and indexers focus on “important” aspects (such as major objects, subject, and relationships) of content (“aboutness”) contained in an image. It is not surprising to find that different people have different interpretations of the “aboutness” of an image; the interpretation of images by catalogers and indexers are not always consistent.

Developed in the early 1990s, Content-Based Image Retrieval (CBIR) uses automatic extraction of lower-level image features (such as texture, color, shape, and structure) to catalog and retrieve images. CBIR classifies and searches images according to similarities in automatically extracted visual features, and output is usually a ranked list of images in order of their similarity to the query

A number of CBIR systems have been developed over the past ten years, including IBM’s Query by Image Content (QBIC) search engine. The State Hermitage Museum in St. Petersburg, Russia, was an early adopter of QBIC for its digitized image collections. The QBIC system allows users to query image databases based on color percentages, color layout, textures, and shapes occurring in the images. In addition, users can use examples (pictures or sketches drawn by the users) to formulate image queries. Content-based queries can also be supplemented with text and keyword queries.

Although there is great interest in research of this type, it is difficult to evaluate a CBIR system because of “lack of agreement on appropriate effectiveness measures and the inherent difficulty in establishing criteria for image similarity” (Graham, 2004, p. 329). As Jörgensen (1999) has asserted, many CBIR techniques are “computationally possible” (p. 302). However, little research has been done to examine how the search and retrieval performance of these CBIR systems correspond with the visual information needs of actual users (Chen, 2001, p. 260). Accordingly, only a few museums, libraries, and archives have adopted CBIR.2 Nevertheless, compared to labor-intensive text-based information retrieval (TBIR), CBIR techniques are inherently faster, less expensive, and much more objective (Graham, 2004, p. 330). Further, a recent study of end users’ image query behaviors by Chen (2001) has suggested the practicability of CBIR-based tools in the field of art history. In particular, Zhang, Pham, and Li (2004) maintain that the CBIR approach is potentially useful to investigate relationships among paintings based on objective visual facts, rather than subjective interpretations (p. 258).

CBIR and the Chinese Landscape

Previous research by Zhang et al. (2004) in applying CBIR to Chinese paintings has suggested that the two elemental issues in CBIR—feature extraction and similarity measures tend to be domain-specific because image content is identified not only by data but also by context (i.e., domain-specific knowledge). This research demonstrated that a case study of a controlled and well-defined domain of images is useful to validate and further enhance existing CBIR techniques. Therefore, this paper will focus on the application of CBIR to traditional Chinese paintings—specifically, Chinese landscape paintings.

With a long and glorious history, Chinese painting has developed its unique form and style by the manipulation of brushes to apply ink and colors to rice paper or silk. In terms of subject matter, Chinese painting can be categorized into three sets: landscape, bird-and-flower, and figure paintings. Of these, landscape painting is regarded as most important because it was a central topic for the literati painters who produced most of the extant Chinese paintings.

Yuan dynasty hanging scroll painting of a mountain and building
The Simple Restreat, Yuan dynasty (1279-1368)
(c) The Metropolitan Museum of Art
www.metmuseum.org
Ca. 1370; Wang Meng (Chinese, ca. 1308-1385); hanging scroll; ink and color on paper; H. 53 ½ in. (136 cm.), W. 17 ¾ in. (45 cm.); signed: "The Yellow Crane Mountain Woodcutter Wang Meng painted this for the lofty scholar of the Simple Retreat" Ex coll.: C.C. Wang Family; promised gift of the Oscar L. Tang Family

As stated by Zhang et al. (2004), CBIR is potentially an excellent and feasible retrieval mechanism for Chinese landscape painting in terms of its visual features. These artworks attempt to capture the essence rather than the real shape of nature to express the painters’ ideas and feelings. Painters of Chinese landscapes use relatively simple forms and textures, only a few colors, and a small number of brush strokes. A limited number of object types are depicted in Chinese landscape paintings, most commonly mountains, rocks, water, clouds, woods, and trees, and sometimes dwellings, pavilions, bridges, figures, and animals. For each object type, there are usually a small number of variations. Furthermore, composition of these paintings follows certain perspectives and models.

Development of the thesaurus

The shape-and-composition CBIR thesaurus below is intended specifically for use with Chinese landscape paintings dating from the Song to Qing periods (AD 960-1911). Approximately 1,000 paintings were studied to extract the features to be included in the thesaurus. Because there are not many extant Song and Yuan paintings, about 80 percent of these painting samples were dated to the Ming and Qing periods (AD 1368-1911).3 Color and texture, as features, are not yet included in the thesaurus. The shape-and-composition CBIR thesaurus emphasizes discrimination among object types to improve retrieval of relevant images. It adopts not only basic shapes (such as circles, rectangles, and triangles) but also combinations of lines (straight, arc, and wavy) and shapes. Special shapes are developed for those object types that are either unique to Chinese art and culture (such as linglong-shaped rocks, bamboos, and dragon boats), or are a peculiar shape that cannot easily be abstracted into basic shapes (such as birds and clouds).

The thesaurus is designed to facilitate the extraction and indexing of image content data for effective retrieval performance. Although it is domain specific, the approach of developing and classifying the thesaurus may be applicable to CBIR of non-Chinese art images and CBIR in general.

Shape-and-Composition CBIR Thesaurus for Chinese Landscape Paintings (AD 960-1911)

This thesaurus is a controlled CBIR list of conceptual forms and compositions extracted from Chinese landscape art ranging from the Song to Qing dynasties (AD 960-1911). It consists of abstract shapes, lines, and composition templates. The Appendix contains a concise version of the thesaurus without explanatory notes.

Shapes

Shapes can be divided into two main types: basic shapes, and special shapes. Shapes may be used individually or in combinations when necessary.

Basic Shapes

  • circle An image of a circle
  • rectangle An image of a rectangle
  • triangle An image of a triangle
  • right triangle An image of a right triangle
  • ellipse An image of a ellipse
  • semi-ellipse An image of a half an ellipse
  • trapezoid An image of a trapezoid
  • right trapezoid An image of a trapezoid with a right angle

Special Shapes

  • irregular polygon An image of an irregular polygon
  • cloud icon An image of a stylized cloud
  • linglong or pierced and rounded irregular polygon An image of a polygon with a circular shape inside it
  • U-shape An image of a capital letter U
  • S-shape
  • V-shape

Lines

Lines include straight lines, arc lines, and wavy lines. Lines are used in groups or in combination with shapes.

  • straight lines An image of a straight horizontal line
  • arc lines An image of a curved, arcing line
  • wavy lines An image of a wavy line, line a rough sine wave

Composition Templates

Composition templates characterize visual layout structures that were commonly adopted in Chinese landscape painting techniques, and are generally used in combination with shapes and/or lines. Accordingly, the thesaurus is cocomposed of two parts: Part I, which focuses on shapes and lines; and Part II, which provides composition templates.

Part I (shapes and lines), the core of the thesaurus, is arranged alphabetically and hierarchically by object types and their variations that frequently appear in Chinese landscape painting. Object types are divided into two main sets: (A) primary elements, and (B) secondary elements. Primary elements refer to object types that can always be found in Chinese landscape painting, and consist of five categories: clouds, mountains, plants, rocks, and water. Secondary elements refer to object types that occasionally appear in Chinese landscape painting, and comprise four categories: animals, architecture, persons, and transportation facilities. Each category is further divided hierarchically into subcategories.

Part II (composition templates) includes 14 templates, which are classified into two main categories (fully filled and one part). They are also organized in alphabetic and hierarchical order.

Each entry includes: a number indicating its order in the thesaurus; the name of the category/subcategory; a scope note (SN) that explains or defines the category; occasional supplemental notes (Note) attached to subcategories when they are not self-explanatory; and abstractions (shapes/shape combinations/line groups/line and shape combinations/composition templates).4

Part I: Shapes and Lines

A. Primary Elements [SN: This set refers to object types that are always found in Chinese landscape painting. It consists of five categories: clouds (A1), mountains (A2), plants (A3), rocks (A4), and water (A5).]

  • A1. Clouds [SN: This category refers to cloud(s) with contours (A1.1) and without contours (A1.2) in various sizes.]
Clouds in Chinese landscape paintings are very difficult to abstract into basic shapes because they were either delineated randomly out of the artist’s imagination or represented by leaving irregular empty spaces of various sizes. Therefore, a universal cloud icon is used to stand for this category; however, the with contours (A1.1) icon is delineated in solid lines while the icon for without contours (A1.2) is represented with dashed lines to demonstrate their difference.

    • A1.1. With Contours An image of a stylized cloud
    • A1.2. Without Contours An image of a stylized made with dotted lines
  • A2. Mountains [SN: This category refers to two main subcategories of mountains according to the meticulousness of the brushstrokes: distant mountains (A2.1), which are usually painted in ink/color washes and a few brushstrokes; and mountains in a close view (A2.2), which are often meticulously delineated with skylines and texture.5 Similar to the way in which clouds with contours (A1.1) are differentiated from clouds without contours (A1.2), shapes in solid outlines are used to stand for mountains in a close view (A2.2), and shapes in dashed lines to represent distant mountains (A2.1).]

Whether depicted in a close view or in the far distance, a mountain in a Chinese landscape painting is generally composed of peaks and crags. Accordingly, each subcategory of the mountains category is divided into two main types: crags (A2.1.1/A2.2.1) and peaks (A2.1.2/A2.2.2). These two main types are further classified hierarchically into their varieties on the basis of their spatial relationships.

When abstracting their undulating skylines into straight lines, peaks and crags are interpreted as two basic shapes: right trapezoids standing for crags, and triangles for peaks. Both shapes can be used in groups and combinations to represent multiple adjacent (A2.1.1.1/A2.1.2.1) or isolated (A2.1.1.2/A2.1.2.2) crags and peaks.

    • A2.1. Distant Mountains
      • A2.1.1 Crags
        • A2.1.1.1 Adjacent An image drawn in dotted lines of two tall skinny trapezoids which are adjacent
        • A2.1.1.2 Isolated An image drawn in dotted lines of two tall skinny trapezoids which are not next to each other
      • A2.1.2 Peaks
        • A2.1.2.1 Adjacent An image drawn in dotted lines of two equilateral triangles next to each other
        • A2.1.2.2 Isolated
        • A2.1.2.2.1 horizontally
        • A2.1.2.2.2 vertically An image drawn in dotted lines of two equilateral triangles which are seperated vertically
    • A2.2. Mountains in a Close View
      • A2.2.1 Crags
        • A2.2.1.1 Adjacent An image of two tall skinny trapezoids which are adjacent
        • A2.2.1.2 Isolated An image of two tall skinny trapezoids which are not next to each other
      • A2.2.2 Peaks
        • A2.2.2.1 Adjacent An image of two equilateral triangles next to each other
        • A2.2.2.2 Isolated
          • A2.2.2.2.1 horizontally An image of two equilateral triangles which are seperated horizontaly
          • A2.2.2.2.2 vertically An image of two equilateral triangles which are seperated vertically
  • A3. Plants [SN: This category refers to the kinds of plants in Chinese landscape paintings: specifically, various types of trees, reeds, and grass. Accordingly, the plants category is divided into three main subcategories: grass (A3.1), reeds (A3.2), and trees (A3.3).]

Because reeds and grass in Chinese landscape paintings are usually depicted with piles of simple brushstrokes, a group of five straight vertical lines is adopted to symbolize them, with longer lines for reeds and shorter lines for grass to illustrate their difference.

In addition, trees are further divided into two main types: popular (A3.3.1), and uncommon (A3.3.2). Popular species of trees include bamboos (A3.3.1.1), pine trees (A3.3.1.2), and willows (A3.3.1.3). Since popular species of trees are differentiated primarily by their leaves in Chinese landscape painting, abstract shapes of leaves are used to represent them. A generalized tree icon is employed to represent the less-frequently used species of trees in Chinese landscape painting.

    • A3.1 Grass An image of short vertical lines
    • A3.2 Reeds An image of long vertical lines
    • A3.3 Trees
      • A3.3.1 Popular
        • A3.3.1.1 Bamboos An image of three nearly overlapping triangles, like bamboo leaves
        • A3.3.1.2 Pine trees
        • A3.3.1.3 Willows An image of curved lines cascading down from an upper point like a willow tree's leaves
      • A3.3.2 Uncommon An image of a triangle on a small vertical rectangle, a universal symbol for a Christmas Tree
  • A4. Rocks [SN: This category refers to various rocks that are portrayed individually or sometimes with plants (like pine trees and bamboos) on the ground, in a garden, or in water. Rocks that compose mountains are excluded from this category because they are components of mountains and are not independent object types.]

According to their various contours, the rocks category is divided into four subcategories: ellipse (A4.1), irregular polygon (A4.2), linglong6 (pierced and rounded irregular polygon) (A4.3), and rectangle (A4.4).

    • A4.1 Ellipse
    • A4.2 Irregular polygon

[Note: The shape serves as a generalized icon because irregular polygon-shaped rocks in the Chinese landscape painting have many varieties and it is unnecessary to categorize each of them.]

    • A4.3 Linglong (pierced and rounded irregular polygon)

[Note: The shape serves as a generalized icon because linglong-shaped rocks in the Chinese landscape painting were usually rendered out of the artist’s imagination and thus have no standard forms. In addition, the one hole here symbolizes one or multiple holes that may be found on linglong-shaped rocks. The number of holes on such rocks is very arbitrary, so it is unnecessary to specify it.]

    • A4.4 Rectangle
  • A5. Water [SN: This category refers to natural water in various volumes. Different types of water are usually represented by simple brushstrokes or lines in a conceptual sense in Chinese landscape painting.]

According to major line types for water delineation, the water category is divided into two subcategories: non-waterfalls (A5.1), including rivers, springs, lakes, seas, and other forms of water which are usually portrayed in wavy lines; and waterfalls (A5.2), which are often depicted in parallel steep arc lines. Hence, a group of horizontally wavy lines represents all kinds of non-waterfalls (A5.1), while a group of three parallel steep arc lines stands for waterfalls (A5.2).

    • A5.1 Non-Waterfalls An image of horizonatal wazy lines

[Note: Representations of specific non-waterfall types (such as rivers, springs, lakes, and seas) are not differentiated in this subcategory because their brushstrokes are essentially the same in nature. Furthermore, visualization of a certain non-waterfall type is sometimes very subjective in Chinese landscape painting. Divergent illustrations can refer to a same non-waterfall type. Therefore, use of a generalized icon is the best way to represent non-waterfall elements for the purposes of this paper.]

    • A5.2 Waterfalls An image of vertical, slightly curving lines

B. Secondary Elements [SN: This set refers to object types that sometimes appear in a Chinese landscape painting. It contains four categories: animals (B1), architecture (B2), persons (B3), and transportation facilities (B4).]

  • B1. Animals [SN: This cat egory refers to various kinds of animals that usually appear in the Chinese landscape painting. It is composed of two main subcategories: birds (B1.1), and mammals (B1.2).]

The birds subcategory consists of three types specified by their various motions, specifically flying (B1.1.1), sitting on the ground or in the water (B1.1.2), and standing (B1.1.3). A V shape is used to represent flying (B1.1.1) and an S shape is used to represent sitting (B1.1.2). While a combination of an S shape, representing the body and two vertical lines for feet are used to symbolize standing (B1.1.3).

The mammals subcategory is further divided into two types according to the popularity of their appearance in the Chinese landscape painting. The first type, popular (B1.2.1), includes three varieties of mammals most commonly seen in Chinese landscape painting, namely deer (B1.2.1.1), donkey/horse (B1.2.1.2), and waterbuffalo (B1.2.1.3). These mammals are essentially differentiated by description of the head in Chinese landscape painting; therefore, abstract representation of the head is used to symbolize them. The second type, uncommon (B1.2.2), refers to mammals other than those in the popular type (B1.2.1). Since they are seldom found in Chinese landscape paintings, it is very difficult to specify and categorize them. Therefore, they are represented by a generalized shape combination which includes a triangle (head) and a rectangle (body).

    • B1.1 Birds
      • B1.1.1 Flying
      • B1.1.2 Sitting on the ground or in the water
      • B1.1.3 Standing An image of a capital letter S with two vertical lines at the bottom (feet)
    • B1.2 Mammals
      • B1.2.1 Popular

[Note: Since the three types of popular mammals are mainly differentiated by representations of the head (including ears and horns), abstract shapes of their head and horns or ears are used to symbolize them.]

        • B1.2.1.1 Deer An image of a triangle, point down, with two lines angling in opposite directions from the top to give the impression of antlers

[Note: Two oblique lines are used to stand for the horns and one triangle for the head.]

        • B1.2.1.2 Donkey/Horse An image of a oval, vertically oriented, with two lines angling in opposite directions from the top to give the impression of antlers

    [Note: Two oblique lines are used to stand for the ears and one ellipse for the head.]

        • B1.2.1.3 Water Buffalo An image of a triangle, point down, with two curved lines angling in opposite directions from the top to give the impression of horns

[Note: Two curves are used to stand for the horns and one triangle for the head.]

      • B1.2.2 Uncommon An image of a triangle, point up, slightly overlapping with a rectangle, to give an impression of a body and a head on an animal
  • B2. Architecture [SN: This category includes two main subcategories: bridges and buildings.]

The bridges subcategory (B2.1) is classified into two types: arch bridges (B2.1.1), and beam bridges (B2.1.2), according to the shape of the bridge deck. A bridge is abstracted into a combination of a rainbow shape (arch), a rectangle (beam) standing for the bridge deck, and two vertical rectangles for all bridge piers.

The buildings subcategory (B2.2), including dwellings, pavilions, and pagodas, is classified into two types according to the number of stories: multi-storied (B2.2.1), and single-storied (B2.2.2). A trapezoid is used for the roof and a rectangle for the building frame.

    • B2.1. Bridges
      • B2.1.1 Arch An image of a stylized arch with a curved top
      • B2.1.2 Beam An image of a stylized arch with a flat top
    • B2.2. Buildings
      • B2.2.1 Multi-storied
      • B2.2.2 Single-storied An image of a block drawn in three dimensions
  • B3. Persons [SN : This category refers to all person(s) in motion (B3.1) and various positions (B3.2).]

The first subcategory includes bending (B3.1.1) and walking (B3.1.2). The second subcategory is divided into three types: lying down (B3.2.1),sitting (B3.2.2), and standing (B3.2.3). As both subcategories walking and standing are abstracted into a same-shape combination, the former is represented with a dashed line and the latter with a solid line to demonstrate the difference. The sitting category is further divided according to the face orientation. A single person is interpreted as composition of two basic shapes, namely a circle standing for the head, and a rectangle or triangle for the body. The rectangle is used when the body is stretching (vertically or horizontally). Otherwise, different types of triangle are used to represent the body.

    • B3.1 In motion
      • B3.1.1 Bending An image of a circle next to a right triangle, arranged roughly in the shape of a person bending over
      • B3.1.2 Walking An image of a circle on top of a vertically aligned rectangle
    • B3.2 In various positions
      • B3.2.1 Lying down An image of a circle next to a horizontally aligned rectangle
      • B3.2.2 Sitting
        • B3.2.2.1 Facing front An image of a circle on top of an equilateral triangle
        • B3.2.2.2 Facing left An image of a circle on top of a right triangle, the hypotenuse of the triangle on the left
        • B3.2.2.3 Facing right An image of a circle on top of a right triangle, the hypotenuse of the triangle on the right
      • B3.2.3 Standing An image of a circle on top of a vertically aligned rectangle
  • B4. Transportation Facilities [SN: This category consists of two main subcategories: boats (B4.1), and carriages (B4.2).]

The boats subcategory (B4.1) includes five types according to their various forms: canoes, dragon boats, fishing boats, passenger ships, and sailing boats. The canoe (B4.1.1) is interpreted as one semi-ellipse that symbolizes the body. The dragon boat (B4.1.2) is unique to China and thus interpreted as a special U shape. The fishing boat (B4.1.3), usually with an oval canopy, is represented by a combination of a smaller semi-ellipse (‘canopy’) and a bigger inverted semi-ellipse (‘body’). The passenger ship (B4.1.4), commonly with a rectangular canopy, is interpreted as one semi-ellipse standing for the body and one rectangle for the canopy. The sailing boat (B4.1.5) is divided into two varieties according to the form of its sail: trapezoid (B4.1.5.1), or triangle (B4.1.5.2). Therefore, a sailing boat is abstracted into a combination of a trapezoid/triangle standing for the sail with a semi-ellipse circle for the body.

A carriage (B4.2) is symbolized by a combination of a circle standing for the wheels and a rectangle for the body.

    • B4.1. Boat
      • B4.1.1. Canoe An image of the lower half of a horizontaly aligned elipse
      • B4.1.2 Dragon boat
      • B4.1.3 Fishing boat An image of the lower half of a horizontaly aligned elipse with half a circle on top of it
      • B4.1.4 Passenger ship An image of the lower half of a horizontaly aligned elipse with a rectangle on top of it
      • B4.1.5 Sailing boat
        • B4.1.5.1 Trapezoid sail An image of the lower half of a horizontaly aligned elipse with a trapezoid on top of it
        • B4.1.5.2 Triangle sail An image of the lower half of a horizontaly aligned elipse with a triangle on top of it
    • B4.2 Carriage

Part II: Composition Templates

This part consists of 1 4 composition templates that are commonly applied to a Chinese landscape painting. Based on the overall layout of objects in a given Chinese landscape painting, t he composition templates are classified into two main categories: fully filled (C1), and one part (C2). Abstract rectangles are used to symbolize objects in the painting and illustrate their spatial relationships.

A Chinese landscape painting in the fully filled (C1) framework represents objects that cover most or all of the rice paper or silk. In terms of spatial relationships between objects, this category is further divided into two subcategories: non-symmetrical (C1.1), and symmetrical (C1.2). The non-symmetrical subcategory includes four types: extended (C1.1.1), fragmented (C1.1.2), vertically superimposed (C1.1.3), and zigzagged (C1.1.4). The extended type is further classified into two varieties: horizontally (C1.1.1.1), and vertically (C1.1.1.2). According to the orientation of balanced parts, the symmetrical subcategory consists of three types: bilateral (C1.2.1), diagonal (C1.2.2), and up and down (C1.2.3).

In contrast to the fully filled composition template, Chinese landscape paintings in the one part (C2) framework represent objects that lie within a specific part of the rice paper or silk. To further identify the location of these objects within the painting, this category contains six subcategories: center (C2.1), left (C2.2), lower (C2.3), lower left (C2.4), lower right (C2.5), and right (C2.6).

  • C1. Fully filled
    • C1.1 Non-symmetrical
      • C1.1.1 Extended
        • C1.1.1.1 Horizontally An image of the a rectangle with three smaller rectangles inside it. The smaller rectangles are next to each other
        • C1.1.1.2 Vertically An image of the a rectangle with three smaller rectangles inside it. The smaller rectangles are on top of each other
        • C1.1.2 Fragmented An image of the a rectangle with three smaller rectangles inside it. The smaller rectangles are arranged randomly

[Note: Objects are spread all over the painting without evident spatial relationships between one another.]

        • C1.1.3 Vertically superimposed An image of the a rectangle with three smaller rectangles inside it. The smaller rectangles overlap each other and are of different heights

[Note: Objects are usually lofty and vertically superimposed.]

        • C1.1.4 Zigzagged An image of the a rectangle with a zig-zagged line drawn inside it

[Note: Objects are arranged in a zigzag route.]

      • C1.2 Symmetrical
        • C1.2.1 Bilateral An image of the a rectangle with two smaller rectangles inside it. The smaller rectangles mirror each other over a vertical axis
        • C1.2.2 Diagonal An image of the a rectangle with two smaller rectangles inside it. The smaller rectangles mirror each other over a diagonal axis sloping right to left or An image of the a rectangle with two smaller rectangles inside it. The smaller rectangles mirror each other over a diagonal axis sloping left to right
        • C1.2.3 Up and down An image of the a rectangle with two smaller rectangles inside it. The smaller rectangles mirror each other over a horizontal axis
    • C2. One part
      • C2.1 Center An image of the a rectangle with one smaller rectangle inside it. The smaller rectangle is centered in the larger one
      • C2.2 Left An image of the a rectangle with one smaller rectangle inside it. The smaller rectangle is left-justified in the larger one
      • C2.3 Lower An image of the a rectangle with one smaller rectangle inside it. The smaller rectangle is on the lowef half of the larger one
      • C2.4 Lower left An image of the a rectangle with one smaller rectangle inside it. The smaller rectangle is in the lower-left corner of the larger one
      • C2.5 Lower right An image of the a rectangle with one smaller rectangle inside it. The smaller rectangle is in the lower right corner of larger one
      • C2.6 Right An image of the a rectangle with one smaller rectangle inside it. The smaller rectangle is right-justified in the larger one

Testing the Thesaurus

An image of mountains in summer, from the Northern Song Dynasty, with trees, a lake, and sailing boats in the forground. The image is in shades of brown and black.
Fig. 1. Summer Mountains, Northern Song (960-1127)
(c) The Metropolitan Museum of Art
www.metmuseum.org
11th century. Attributed to Qu Ding (Chinese, active ca. 1023-ca.1056); ex coll.: C.C. Wang Family; gift of the Dillon Fund, 1973

To test the ability of the thesaurus to adequately represent the content of the paintings, it was applied to four randomly selected Chinese landscape paintings, which are dated to different time periods and had not been used previously in developing the thesaurus. The features and composition templates allowed adequate representation of the content of the paintings. One example is reported below.7 No attempt was made to test retrieval, which would require access to an entire collection indexed with the thesaurus. Nevertheless, users may run a query by selecting abstract shapes and/or compositions if they wish to find landscapes which bear certain shapes and/or compositions.

In Figure 1, the Song landscape consists of (A) primary elements (mainly peaks and crags in both the distance and in close view, pine trees, and a river), and (B) secondary elements (specifically sailing boats). The counterparts in the thesaurus are:

        • A2.1.1.2 Isolated (distant crags)
        • A2.1.2.1 Adjacent (distant peaks)
          • A2.1.2.2.1 horizontally (isolated distant peaks)
          • A2.1.2.2.2 vertically (isolated distant peaks)
        • A2.2.1.2 Isolated (close crags)
        • A2.2.2.1 Adjacent (close peaks)
          • A2.2.2.2.1 horizontally (isolated close peaks)
          • A2.2.2.2.2 vertically (isolated close peaks)
      • A3.3.1.2 Pine trees
    • A5.1 Non-Waterfalls
        • B4.1.5.2 Triangle sail (sailing boat)

The composition of this Song landscape corresponds to C1.1.1.1 (fully-filled, non-symmetrical, and horizontally extended).

Conclusion and Future Work

This paper develops a shape-and-composition CBIR thesaurus for traditional Chinese landscape painting dating from the Song to Qing periods (960-1911). The thesaurus is based on visual features of Chinese landscape paintings, including less complexity of forms and textures, a few colors, and a certain number of object types, varieties, and composition structures. Not only basic shapes (such as circle, triangle, and rectangle) but also lines and shape combinations are adopted in the thesaurus. Furthermore, special shapes are developed to represent object types with unique forms. By emphasizing discrimination among object types, this thesaurus aims to improve recall of relevant images. Results from testing the thesaurus demonstrate that shape features and composition templates are sufficient to represent the content of the paintings. Therefore, this shape-and-composition CBIR thesaurus has the potential to be a feasible and effective means to index and retrieve Chinese landscape paintings. It may be applied to those museums and libraries that have large collections of these works, such as the Freer Gallery of Art and the Metropolitan Museum of Art. As mentioned previously, Chinese landscape paintings were created primarily to convey the painters’ ideas and feelings. Textual interpretations of the hidden meanings in the paintings are inherently subjective and partial. The perceived objectivity of CBIR may be useful to discover new knowledge that has never been studied or was actually misinterpreted.

Further research includes processing additional paintings to test and ensure the completeness of the thesaurus with regard to shapes and composition abstractions in Chinese landscape paintings. Although numerous paintings were studied to develop the features in the thesaurus and the most common have been included, an even larger number of sample paintings may need to be processed. In addition, it may be necessary to develop some computing algorithm and techniques to automatically process the feature extraction. Furthermore, to make the thesaurus more solid and effective for a CBIR system, the thesaurus needs to incorporate color and texture as well. Finally, the thesaurus should be tested to determine its usefulness for adequately representing users’ information needs and its retrieval effectiveness in differentiating among various elements of Chinese landscape paintings.

Acknowledgement

I would like to express many thanks to Professor Marilyn Domas White and Ms. Joan Stahl for their helpful advice and generous support. I would also like to thank Yu-tzu Chang, a former classmate who did the initial research and report with me.

References

Chen, H. (2001). An analysis of image queries in the field of art history. Journal of the American Society for Information Science and Technology,52(3), 260-273.

Graham, M. E. (2004). Enhancing visual resources for searching and retrieval—Is content-based image retrieval a solution? Literary & Linguistic Computing: Journal of the Association for Literary and Linguistic Computing, 19(3), 321-333.

Jörgensen, C. (1999). Access to pictorial material: A review of current research and future prospects. Computers and the Humanities, 33, 293-318.

Lu, F., & Shen, M. (1990). Zhong guo hua li dai ming jia ji fa tu pu, shan shui bian (Vols.1-8) . Shanghai: shanghai shu hua chubanshe.

Siren, O. (1973). Chinese painting: Leading masters and principles (Vols.1-2). New York: Hacker Art Books.

Wu, Y. (1990). The techniques of Chinese painting. London: The Herbert Press.

Yang, C. C. (2004). Content-based image retrieval: A comparison between query by example and image browsing map approaches. Journal of Information Science, 30(3), 254-267.

Zhang, D., Pham, B., & Li, Y. (2004). Modeling traditional Chinese paintings for content-based image classification and retrieval. Proceedings of the 10th International Multimedia Modeling Conference, 258-64.

Notes

1 This paper is a revised and shortened version of a term paper written in spring 2006. The original paper (approximately 48 pages), complete with illustrations from Chinese landscapes, is posted at: http://shinylee. pages.com/CBIRThesaurusforChineseLandscape_Tan.pdf.

2Aside from the Heritage Museum, other image collections using CBIR system include the Leiden 19th-Century Portrait Database from Leiden University, The Netherlands, and The SCULPTEUR (Semantic and content-based multimedia exploitation for European benefit) Project from The Victoria & Albert Museum, U.K.

3 Due to limited time, no statistics for the sample set distributed by historical period has been developed.

4Chinese landscape examples with superimposed abstractions are not provided here because of limited space but are available in the original, lengthier version of the paper posted at: http://shinylee. pages.com/CBIRThesaurusforChineseLandscape_Tan.pdf

5It should be noted that the meticulousness of brushstrokes to delineate a mountain is a key measurement to classify whether a mountain is a ‘close view’ or ‘distant view.’ This is because Chinese landscapes were painted in a unique many-point instead of one-point perspective. For example, behind a mountain range, another range may loom with trees, houses, and streams in full view. As mentioned previously, Chinese landscapes advocate to capture the essence rather than the reality of the nature.

6 Linglong -shaped rocks are unique to the Chinese culture. They were once very popular in ancient Chinese gardens. Their grotesque forms were mostly carved by Chinese artists or artisans.

7Other examples are available in the original paper posted at http://shinylee. pages.com/CBIRThesaurusforChineseLandscape_Tan.pdf

Appendix: Concise Shape-and-Composition CBIR Thesaurus

Part I: Shapes and Lines

A. Primary Elements

  • A1. Clouds
    • A1.1. With Contours
    • A1.2. Without Contours
  • A2. Mountains
    • A2.1. Distant Mountains
      • A2.1.1 Crags
        • A2.1.1.1 Adjacent
        • A2.1.1.2 Isolated
      • A2.1.2 Peaks
        • A2.1.2.1 Adjacent
        • A2.1.2.2 Isolated
        • A2.1.2.2.1 horizontally
        • A2.1.2.2.2 vertically
    • A2.2. Mountains in a Close View
      • A2.2.1 Crags
        • A2.2.1.1 Adjacent
        • A2.2.1.2 Isolated
      • A2.2.2 Peaks
        • A2.2.2.1 Adjacent
        • A2.2.2.2 Isolated
          • A2.2.2.2.1 horizontally
          • A2.2.2.2.2 vertically
  • A3. Plants
    • A3.1 Grass
    • A3.2 Reeds
    • A3.3 Trees
      • A3.3.1 Popular
        • A3.3.1.1 Bamboos
        • A3.3.1.2 Pine trees
        • A3.3.1.3 Willows
      • A3.3.2 Uncommon
  • A4. Rocks
    • A4.1 Ellipse
    • A4.2 Irregular polygon
    • A4.3 Linglong (pierced and rounded irregular polygon)
    • A4.4 Rectangle
  • A5. Water
    • A5.1 Non-Waterfalls
    • A5.2 Waterfalls

B. Secondary Elements

  • B1. Animals
    • B1.1 Birds
      • B1.1.1 Flying
      • B1.1.2 Sitting on the ground or in the water
      • B1.1.3 Standing
    • B1.2 Mammals
      • B1.2.1 Popular
        • B1.2.1.1 Deer
        • B1.2.1.2 Donkey/Horse
        • B1.2.1.3 Water Buffalo
      • B1.2.2 Uncommon
  • B2. Architecture
    • B2.1. Bridges
      • B2.1.1 Arch
      • B2.1.2 Beam
    • B2.2. Buildings
      • B2.2.1 Multi-storied
      • B2.2.2 Single-storied
  • B3. Persons
    • B3.1 In motion
      • B3.1.1 Bending
      • B3.1.2 Walking
    • B3.2 In various positions
      • B3.2.1 Lying down
      • B3.2.2 Sitting
        • B3.2.2.1 Facing front
        • B3.2.2.2 Facing left
        • B3.2.2.3 Facing right
      • B3.2.3 Standing
  • B4. Transportation Facilities
    • B4.1. Boat
      • B4.1.1. Canoe
      • B4.1.2 Dragon boat
      • B4.1.3 Fishing boat
      • B4.1.4 Passenger ship
      • B4.1.5 Sailing boat
        • B4.1.5.1 Trapezoid sail
        • B4.1.5.2 Triangle sail
    • B4.2 Carriage

Part II: Composition Templates

  • C1. Fully filled
    • C1.1 Non-symmetrical
      • C1.1.1 Extended
        • C1.1.1.1 Horizontally
        • C1.1.1.2 Vertically
        • C1.1.2 Fragmented
        • C1.1.3 Vertically superimposed
        • C1.1.4 Zigzagged
      • C1.2 Symmetrical
        • C1.2.1 Bilateral
        • C1.2.2 Diagonal or
        • C1.2.3 Up and down
    • C2. One part
      • C2.1 Center
      • C2.2 Left
      • C2.3 Lower
      • C2.4 Lower left
      • C2.5 Lower right
      • C2.6 Right

Author's Bio

Tang Li recently received an MLS from the College of Information Studies, University of Maryland (UM), College Park. She is also a graduate assistant of UM art and architecture libraries. Her research interests are in art librarianship, reference, image searching and retrieval, and collection management.

Go to Top

Contents

  1. Editor's Note
  2. Abstract
  3. Introduction
  4. Content-Based Image Retrieval (CBIR) vs. Text-Based Image Retrieval (TBIR)
  5. CBIR and the Chinese Landscape
  6. Development of the thesaurus
  7. Shape-and-Composition CBIR Thesaurus for Chinese Landscape Paintings (AD 960-1911)
  8. Testing the Thesaurus
  9. Conclusion and Future Work
  10. Acknowledgement
  11. References
  12. Notes
  13. Appendix:Concise Shape-and-Composition CBIR Thesaurus
  14. Author's Bio