Chapter 10 Group clustering

How to group sites according to dissimilarity of community compositions?

10.1 Hierarchical: Ward’s clustering

Ward’s clustering is a hierarchical cluster analysis method based on community dissimilarities (Murtagh and Legendre 2014). In this approach, we can set the number of groups a priori for visualization and assigning group membership.

k   <- 7                                          # specify number of groups
cl  <- hclust(D, method='ward.D2')                # clustering solution
grp <- cutree(cl, k)                              # group memberships
plot(cl) ; rect.hclust(cl, k, border=rainbow(k))  # plot the dendrogram

# save cl as cl_ward to use below
cl_ward <- cl

NOTE: Height represents the amount of within-cluster variation introduced when clusters are merged. Clusters that join at low heights are very similar, whereas clusters that join at high heights are much more distinct.

10.2 Non-hierarchical: fuzzy clustering

Non-hierarchical methods also exist. And, it may be useful to define groups based on fuzzy rather than crisp partitions, using fuzzy clustering (de Caceres et al. 2010).

# install.packages("vegclust")
require(vegclust)
## Loading required package: vegclust
ns   <- 1       # number of random starts (increase for real analyses!)
k    <- 7       # number of groups is specified in advance
# fuzzy c-means
cl   <- vegclustdist(D, mobileMemb=k, method='FCM', m=1.2, nstart=ns)
# fuzzy c-means with a noise cluster
cl_a <- vegclustdist(D, mobileMemb=k, method='NC', m=1.2, dnoise=0.8, nstart=ns)
# fuzzy c-medoids
cl_b <- vegclustdist(D, mobileMemb=k, method='FCMdd', m=1.2, nstart=ns)
# crisp k-means
cl_c <- vegclustdist(D, mobileMemb=k, method='KM', nstart=ns)

# how well do the other methods agree with fuzzy c-means?
concordance(cl, cl_a)
## adjustedRand 
##    0.8224128
concordance(cl, cl_b)
## adjustedRand 
##    0.6162529
concordance(cl, cl_c)
## adjustedRand 
##    0.5436687

Let’s examine group memberships grp from the fuzzy clustering. While memberships are fuzzy, we can ask for a crisp membership based on a threshold or else the maximum fuzzy membership.

head(round(cl$memb, 2))               # fuzzy membership
##     M1   M2   M3   M4  M5   M6   M7
## 1 0.00 0.97 0.01 0.00 0.0 0.00 0.01
## 2 0.00 0.00 0.00 0.89 0.1 0.01 0.00
## 3 0.00 0.00 0.00 0.10 0.9 0.00 0.00
## 4 0.00 0.00 0.00 0.00 0.0 0.04 0.96
## 5 0.03 0.02 0.00 0.00 0.0 0.64 0.31
## 6 0.00 0.00 0.00 0.00 0.0 1.00 0.00
(grp <- defuzzify(cl, 'cut', alpha=0.8)[[2]]) # crisp, at threshold (incl NAs)
##    1    2    3    4    5    6    7    8    9   10   11   12   13   14   15   16 
## "M2" "M4" "M5" "M7"   NA "M6" "M6" "M2" "M5" "M2" "M2" "M7" "M2" "M2" "M2" "M7" 
##   17   18   19   20   21   22   23   24   25   26   27   28   29   30   31   32 
## "M7" "M5" "M2" "M2"   NA "M5" "M5" "M5" "M5" "M5" "M2" "M5"   NA   NA "M5" "M5" 
##   33   34   35   36   37   38   39   40   41   42   43   44   45   46   47   48 
## "M7" "M5"   NA "M6" "M7" "M6" "M2" "M5" "M5" "M5" "M4" "M4" "M4" "M4" "M5" "M5" 
##   49   50   51   52   53   54   55   56   57   58   59   60   61   62   63   64 
## "M4" "M4" "M4"   NA "M5" "M5" "M5" "M5" "M5" "M5" "M3" "M5" "M6" "M6" "M1" "M1" 
##   65   66   67   68   69   70   71   72   73   74   75   76   77   78   79   80 
## "M1" "M2" "M6" "M2" "M6"   NA "M3" "M3" "M3" "M3" "M3" "M1" "M3" "M3"   NA "M2" 
##   81   82   83   84   85   86   87   88   89   90   91   92   93   94   95   96 
## "M3" "M1" "M1" "M1" "M1" "M1" "M1" "M2" "M1" "M1" "M7" "M7" "M1" "M1" "M1" "M1" 
##   97 
## "M1"
(grp <- defuzzify(cl, 'max')[[2]])    # crisp membership, at max membership
##    1    2    3    4    5    6    7    8    9   10   11   12   13   14   15   16 
## "M2" "M4" "M5" "M7" "M6" "M6" "M6" "M2" "M5" "M2" "M2" "M7" "M2" "M2" "M2" "M7" 
##   17   18   19   20   21   22   23   24   25   26   27   28   29   30   31   32 
## "M7" "M5" "M2" "M2" "M2" "M5" "M5" "M5" "M5" "M5" "M2" "M5" "M7" "M2" "M5" "M5" 
##   33   34   35   36   37   38   39   40   41   42   43   44   45   46   47   48 
## "M7" "M5" "M4" "M6" "M7" "M6" "M2" "M5" "M5" "M5" "M4" "M4" "M4" "M4" "M5" "M5" 
##   49   50   51   52   53   54   55   56   57   58   59   60   61   62   63   64 
## "M4" "M4" "M4" "M5" "M5" "M5" "M5" "M5" "M5" "M5" "M3" "M5" "M6" "M6" "M1" "M1" 
##   65   66   67   68   69   70   71   72   73   74   75   76   77   78   79   80 
## "M1" "M2" "M6" "M2" "M6" "M6" "M3" "M3" "M3" "M3" "M3" "M1" "M3" "M3" "M1" "M2" 
##   81   82   83   84   85   86   87   88   89   90   91   92   93   94   95   96 
## "M3" "M1" "M1" "M1" "M1" "M1" "M1" "M2" "M1" "M1" "M7" "M7" "M1" "M1" "M1" "M1" 
##   97 
## "M1"
table(grp, useNA='always')            # tally points per group
## grp
##   M1   M2   M3   M4   M5   M6   M7 <NA> 
##   18   17    9    9   25   10    9    0

Compare fuzzy clusters to Ward clusters

table(grp, cutree(cl_ward, k = 7))
##     
## grp   1  2  3  4  5  6  7
##   M1  0  0  0  1  1  0 16
##   M2 16  0  0  0  1  0  0
##   M3  0  0  0  0  8  1  0
##   M4  0  9  0  0  0  0  0
##   M5  0  0 23  0  0  2  0
##   M6  0  0  0 10  0  0  0
##   M7  0  0  0  2  5  0  2

The Ward and fuzzy clustering solutions show substantial agreement, with several fuzzy clusters corresponding closely to individual Ward clusters. However, some Ward groups, especially clusters 5, 6, and 7, are split across multiple fuzzy clusters, suggesting that these communities may represent transitional or less clearly separated assemblages.

10.3 Key references

de Caceres, M., X. Font, F. Oliva. 2010. The management of numerical vegetation classifications with fuzzy clustering methods. Journal of Vegetation Science 21: 1138–1151.

Murtagh, F., and P. Legendre. 2014. Ward’s hierarchical agglomerative clustering method: which algorithms implement Ward’s criterion? Journal of Classification 31: 274–295.