The question of unitarity of representations in the analytic continuation of
discrete series from a Borel--de Siebenthal chamber is considered for those
linear equal-rank classical simple Lie groups G
that have not been treated fully before.
Groups treated earlier by other authors include
those for which G has real
rank one or has a symmetric space with an invariant complex structure.
Thus the groups in question are locally isomorphic to SO(2m,n)_0 with
m \geq 2 and n \geq 3, or to Sp(m,n) with m \geq 2 and n \geq 2.
The representations under study are obtained from cohomological induction.
One starts from a
finite-dimensional irreducible representation of a compact subgroup L of G
associated to a Borel--de Siebenthal chamber, forms a generalized
Verma-like module, applies a derived Bernstein functor, and passes to a
specific irreducible quotient.
Enright, Parthasarthy, Wallach, and Wolf had previously identified all cases
where the representation of L is 1-dimensional and the generalized
Verma-like module is irreducible; for these cases they proved that unitarity
is automatic.
B.~Gross and Wallach had proved unitarity for additional cases for a
restricted class of groups when the representation of L is 1-dimensional.
The present work gives results for all groups and allows higher-dimensional
representations of L.
In the case of 1-dimensional representations of L, the results address
unitarity and nonunitarity and are conveniently summarized in a table that
indicates how close the results are to best possible.
In the case of higher-dimensional representations of L, the method
addresses only unitarity and in effect proceeds by reducing matters to
what happens for a 1-dimensional representation of L and a
lower-dimensional group G.