The question of unitarity of representations in the analytic continuation of
discrete series from a Borel--de Siebenthal chamber is considered for those
linear equal-rank classical simple Lie groups G
that have not been treated fully before.
Groups treated earlier by other authors include
those for which G has real
rank one or has a symmetric space with an invariant complex structure.
Thus the groups in question are locally isomorphic to SO(2m,n)_0 with
m \geq 2 and n \geq 3, or to Sp(m,n) with m \geq 2 and n \geq 2.
The representations under study are obtained from cohomological induction. One starts from a finite-dimensional irreducible representation of a compact subgroup L of G associated to a Borel--de Siebenthal chamber, forms a generalized Verma-like module, applies a derived Bernstein functor, and passes to a specific irreducible quotient. Enright, Parthasarthy, Wallach, and Wolf had previously identified all cases where the representation of L is 1-dimensional and the generalized Verma-like module is irreducible; for these cases they proved that unitarity is automatic. B.~Gross and Wallach had proved unitarity for additional cases for a restricted class of groups when the representation of L is 1-dimensional.
The present work gives results for all groups and allows higher-dimensional representations of L. In the case of 1-dimensional representations of L, the results address unitarity and nonunitarity and are conveniently summarized in a table that indicates how close the results are to best possible. In the case of higher-dimensional representations of L, the method addresses only unitarity and in effect proceeds by reducing matters to what happens for a 1-dimensional representation of L and a lower-dimensional group G.