We propose a method based on the creation of a new image modality consisting in a grayscale map where the value of each pixel indicates its probability of belonging to a cell nuclei. This probability map is calculated from texture and scale information in addition to simple pixel color intensities. The resulting modality has a strong object-background contrast and evens out the irregularities within the nuclei or the background. The actual extraction is performed using an AC model with a nuclei shape prior included to deal with overlapping nuclei.

### Feature model

First, a color deconvolution [1] is applied in order to separate the immunohistochemical stains from which 3 grayscale images are produced: a haematoxilin image, an eosin image and a third residual component orthogonal in RGB space. Next, local features based on Laws’ texture measures [2] are computed for each pixel of the 3 obtained images. 5 different 1-dimensional convolution kernels (L5 = (*1*, *4*, *6*, *4*, *1*), *E*
_{
5
} = (*–1*, *–2*, *0*, *2*, *1*), *W*
_{
5
} = (*–1*, *2*, *0*, *–2*, *1*), *S*
_{
5
} = (*–1*, *0*, *2*, *0*, *–1*) and *R*
_{
5
} = (*1*, *–4*, *6*, *–4*, *1*)) are used to compute 25 different 5 × 5 kernels by convolving avertical 1-dimensional kernel with a horizontal one. The 5 × 5 kernels are applied at every pixel to extract 25 features which are then combined into 15 rotationally invariant features after normalizing by the output of the *L*
_{
5
}
^{
T
} × *L*
_{
5
} kernel and smoothing with a Gaussian kernel of standard deviation *σ* = *1.5* pixels.

The same process is repeated at 4 different scales after locally re-sampling the image using Lanczos-3 sinc kernels. Re-sampled images are locally computed around each pixel to allow the computation of the 15 texture features for the same pixel at different scales. Local texture features are computed at 1:1, 1:2, 1:4 and 1:8 scales for every pixel.

### Probability map

The resulting 180-dimensional feature vector

*x* is used to compute the probability

*p*
_{
n
}(

*x*) of each pixel to belong to a cell nuclei. Let

*μ*
_{
n
} (resp.

*μ*
_{
b
}) be the mean of the feature vectors for the pixels belonging to the nuclei (resp. to the background). A class dependent LDA is performed in order to find two directions in the feature space,

*w*
_{
n
} and

*w*
_{
b
}, such that the projection of the classes on these directions has a maximum inter-class scatter over within-class scatter ratio. The estimated class probability associated with the feature vector x is then calculated from the linear scores

*l*
_{
n
} = (

*x – μ*
_{
n
})

*· w*
_{
n
} and

*l*
_{
b
} = (

*x – μ*
_{
b
})

*· w*
_{
b
} using the softmax function:

The resulting probability map exhibits strong contrast between the objects and the background. Moreover, nuclei and background appear more homogeneous than in the original image. A post processing step is also applied to fill small holes still remaining in nuclei (larger holes are not removed to prevent the unintended deletion of interstices between different nuclei).

### AC model including shape prior

The actual extraction of cell nuclei is performed from the probability map with an AC model with shape prior information. The total energy *E*(*γ*) associated to a contour γ is a weighted sum of an image term *E*
_{
i
}(*γ*) and a shape term *E*
_{
s
}(*γ*). The latter is itself the weighted sum of a smoothing term *E*
_{
sm
}(*γ*) and a shape prior term *E*
_{
sp
}(*γ*).

The shape prior term
allows to control the perturbations *δr*(*t*) of a contour around a circle at different frequencies *k* of the Fourier components by adjusting the coefficients *f*
_{
k
}. Detailed formulas and explanations for this and the other energy terms can be found in the work of Kulikova *et al.* [3]. The shape prior information allows to properly extract overlapping nuclei according to their expected shape without arbitrarily discarding the overlapping parts.

The detection of nuclei is performed by a marked point process model the details about which the interested reader can find in [4]. An empirical study in [5] shows that this particular combination of MPP and AC over-performs other state-of-the-art methods for nuclei detection and extraction.